AI Evaluation Basics Quiz

Answers and explanations appear as you take the quiz.

1. What should a team define before evaluating an AI system?
   • Acceptance criteria for inputs, expected outputs, tolerable errors, and escalation
   • A press release about model quality
   • A single average score
   • A deployment date only
2. How should development examples differ from a held-out evaluation set?
   • Use the held-out set for prompt tuning every day
   • Use development examples for iteration and reserve held-out examples to test choices
   • Mix both sets before every run
   • Keep no record of which examples were used
3. What happens when a team repeatedly tunes on its final test set?
   • The test set becomes larger
   • The score becomes a baseline
   • The test set becomes another development set
   • The model automatically generalizes better
4. Which comparison can show whether added model complexity provides practical benefit?
   • Compare only with last week’s best score
   • Compare with a simple baseline
   • Compare against a random deployment date
   • Compare with a different task’s benchmark
5. For a summarizer, which evaluation checks does the guide recommend?
   • Server uptime and account age
   • Pixel similarity and frame rate
   • Factual consistency and coverage
   • Number of training epochs
6. How should a team verify an AI agent’s completed action?
   • Trust its success message
   • Count how many tools it called
   • Check the authoritative final state and unintended side effects
   • Ask the same model whether it succeeded
7. Why might a rare failure deserve more attention than many harmless wording differences?
   • Consequences of the failure can outweigh frequent low-impact variations
   • Rare events are always measurement errors
   • Wording differences cannot be evaluated
   • The average score must be ignored
8. What should an evaluation document about its limitations?
   • That every real-world case was tested
   • Where the evaluation does not represent actual use
   • Only the name of the model
   • That uncertainty has been eliminated