Prompt Versioning and Regression Testing Quiz

Answers and explanations appear as you take the quiz.

1. Which set of items should be versioned together as one prompt configuration?
   • Only the prompt wording
   • Only the model name
   • Prompt template, model identifier, sampling parameters and tool schemas
   • Only the user inputs from production
2. Why pin a dated model snapshot instead of a generic latest alias?
   • So model changes happen only when you deliberately switch and re-test
   • Because aliases are always slower
   • Because snapshots are free
   • Because aliases do not support system prompts
3. What does the guide say about temperature 0?
   • It guarantees identical outputs every time
   • It makes regression testing unnecessary
   • It only affects the length of outputs
   • It reduces randomness but does not guarantee identical outputs across runs
4. How should a regression suite handle nondeterministic outputs?
   • Require exact string matches
   • Use thresholds and repeated samples instead of exact equality
   • Ignore any failing cases
   • Only test with a single example
5. What is a strong source of new test cases for a golden set?
   • Past production failures turned into test cases
   • Random text from the web
   • Only cases the current prompt already passes
   • Outputs copied from a competitor
6. If an LLM judge grades outputs, what should you do?
   • Let the judge model change freely
   • Trust every verdict without review
   • Pin the judge's model and prompt and check it against human labels
   • Use the same prompt under test as the judge
7. Why log the prompt version ID with each production request?
   • To reduce token costs
   • To improve model accuracy
   • To satisfy the model provider's terms
   • To tie problems to a specific deploy and roll back quickly
8. Why list individual cases that flipped from pass to fail rather than only the average score?
   • Averages are always wrong
   • An unchanged average can hide new breakage offset by gains elsewhere
   • Flipped cases reduce API costs
   • Only flipped cases are graded