Building an LLM Eval Dataset Quiz

Answers and explanations appear as you take the quiz.

1. What does the guide recommend as the best starting source for eval cases?
   • Only LLM-generated synthetic questions
   • Public benchmark leaderboards
   • Random text scraped from the internet
   • Real usage, such as logs, support tickets and search queries
2. When is a rubric more appropriate than a single golden answer?
   • For a classification task with fixed labels
   • For open-ended tasks where many different wordings can be correct
   • For extracting an exact invoice number
   • A rubric is never more appropriate
3. Which of these is a well-written rubric criterion?
   • Is helpful
   • Sounds natural
   • Mentions the refund deadline
   • Is high quality
4. With 100 pass/fail cases and a pass rate near 80 percent, what does the guide say about the margin of error?
   • It is roughly plus or minus 8 percentage points, so a two-point improvement looks like noise
   • It is about plus or minus 0.1 percentage points
   • It is exactly zero
   • It is about plus or minus 50 percentage points
5. What is the purpose of a regression slice?
   • To delete outdated cases automatically
   • To speed up the model's responses
   • To keep past failures in the set so fixes stay tested
   • To train the model on its own mistakes
6. What should you do before trusting an LLM judge to grade rubric items?
   • Raise the judge's temperature
   • Remove the rubric so the judge can decide freely
   • Test the judge on a single case
   • Have people label a sample and measure how often the judge agrees with them
7. Why keep a held-out portion of the dataset that you do not tune against?
   • To avoid overfitting prompts to the test set
   • To save storage space
   • Because regulations require it
   • To reduce latency
8. Why are paired comparisons on the same cases useful when comparing two versions?
   • They remove the need for any grader
   • They usually need fewer examples than comparing independent samples
   • They remove all nondeterminism from model outputs
   • They guarantee a perfect score