Debugging a Prompt That Isn't Working

When a prompt misses the goal, first describe the failure precisely: wrong format, missing detail, unsupported claim, refusal, or poor task completion. Then check whether the prompt states the user’s goal, relevant context, constraints, audience, and desired output. OpenAI’s prompt guidance recommends clarity, specificity, and iterative refinement. Treat prompt changes as small experiments. Keep the original as a baseline, form a hypothesis, change one component, and compare outputs on the same representative test cases. If you change role, examples, output schema, and tone at once, you may not know which change mattered. Record the prompt version and test results. A response can vary across runs, so repeat when sampling or backend variability is relevant. Use concrete checks instead of “better”: required fields present, word limit satisfied, citations supported, or task completed. Include edge cases and examples where the old prompt failed. If outputs remain inconsistent, inspect tool behavior, retrieved context, model version, and system-level instructions—not only the user prompt. For high-stakes tasks, use structured output validation or human review. Prompt changes can improve behavior in the tested setup, but they are not a guarantee for every future input. Keep a holdout set to check whether improvements generalize, and avoid changing the evaluation examples to make a revised prompt look better. When the task has changed, revise the goal explicitly rather than patching around the old request.