LLM Evaluations Quiz

Answers and explanations appear as you take the quiz.

1. What should an LLM evaluation measure?
   • A defined task and its relevant failure conditions
   • Only whether the response sounds fluent
   • The model’s parameter count alone
   • Whether a benchmark is popular
2. Why evaluate the system configuration users actually receive?
   • The base model cannot produce text
   • Prompts, retrieval, tools, and call limits can change behavior
   • The application always scores higher
   • A deployed system cannot be tested
3. When can exact matching be an appropriate evaluation check?
   • For every open-ended answer
   • Only when reviewers disagree
   • For some extracted fields or executable tests
   • To measure response cost
4. What helps different reviewers apply an interpretive rubric consistently?
   • Hide the scoring criteria
   • Use a single preference score
   • Remove examples that are difficult to judge
   • Write the rubric clearly and examine reviewer disagreements
5. How should a team validate a model used to grade outputs?
   • Check it against independently reviewed examples and inspect order or style effects
   • Treat its score as independent proof
   • Ask the same model to approve itself
   • Use only its average score
6. Which cases may be useful to include when relevant to the application?
   • Only short questions with one obvious answer
   • Unanswerable questions, conflicting sources, long-context cases, and malicious retrieved instructions
   • Only prompts used during development
   • No difficult examples
7. How should evaluation results be reported?
   • As one preference number without context
   • Only by model size
   • By task and error severity
   • As a claim that uncertainty is eliminated
8. What should a team do with failed examples?
   • Discard them after calculating a score
   • Add them to prompts until every case passes
   • Treat them as proof of general performance
   • Retain them as regression cases and refresh held-out material over time