LLM-as-a-Judge Quiz

Answers and explanations appear as you take the quiz.

1. What does 'LLM-as-a-judge' refer to?
   • Using one language model to score or compare another model's outputs
   • Using a language model to write legal verdicts
   • A model that only answers yes/no questions
   • A model that ranks search results by date
2. What is the difference between pointwise and pairwise judging?
   • Pointwise is faster; pairwise is slower but identical otherwise
   • Pointwise grades one answer against a rubric; pairwise picks the better of two
   • Pointwise only works for code; pairwise only for prose
   • There is no real difference
3. Which of these is a well-documented bias in LLM judges?
   • Preferring answers written in all capital letters
   • Always preferring the first word alphabetically
   • Favoring longer answers regardless of quality
   • Refusing to grade questions about history
4. How do evaluators commonly counter position bias in pairwise comparisons?
   • By only showing one answer at a time
   • By always putting the longer answer first
   • By deleting the justification field
   • By running each comparison twice with the order swapped
5. What evaluation procedure did G-Eval use for generated text?
   • Chain-of-thought steps followed by a form-filling evaluation against criteria
   • A speaker verification threshold
   • A search index with BM25 only
   • An image classifier trained on screenshots
6. How can a team confirm an LLM judge stays aligned with human preferences?
   • By never checking it again after setup
   • By calibrating it against a human-labeled gold set
   • By making the judge grade itself
   • By only using it on weekends
7. What is self-preference bias in LLM-based evaluation?
   • A judge always selects the shortest answer
   • A judge cannot compare two outputs
   • A judge favors outputs generated by its own model or model family
   • A judge evaluates only audio signals
8. What does G-Eval use to make its discrete score more fine-grained in the reported method?
   • The position of the answer in a pairwise comparison
   • The judge model’s parameter count only
   • A count of citations in the answer
   • Probability weights over the candidate score tokens