Guide-linked quiz · Hard Level
Calibrating LLM Judges Against Humans Quiz
Calibrate LLM judges with shared rubrics, independent human labels, confusion matrices, suitable agreement statistics, and repeated held-out checks.
Question 1 of 8
Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
- Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
- Kappa is always higher than percent agreement
- Kappa does not require human-labeled data
- Kappa can only be used with three or more raters
- What does a confusion matrix show when calibrating an LLM judge against human labels?
- The judge's response latency for each example
- How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
- The total token cost of running the judge
- The order in which examples were labeled
- In the example where a judge rates borderline-acceptable answers as 'excellent,' what does this pattern usually indicate?
- The judge model is fundamentally incapable of grading
- A hardware failure in the evaluation pipeline
- A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
- That human raters made a labeling error
- What can one round of judge calibration fail to account for?
- The model’s parameter count
- The number of UI buttons
- Whether the output includes a period
- The fact that a rubric or model version can change over time
- What does high agreement between an LLM judge and human ratings fail to prove by itself?
- That the rubric is valid or that the labels are correct
- As proof all human labels are correct
- As a reason to skip adjudication
- As a universal threshold for every task
- Why is measuring inter-human agreement useful before evaluating a judge?
- It has no real use in the calibration process
- As a useful reference point, not a universal cap or proof of correctness
- It replaces the need for a confusion matrix
- It determines the judge's token budget
- According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
- Hide the disagreements in an aggregate score
- Delete the category from every dataset
- Inspect those examples, clarify the category rubric, and retest on held-out cases
- Assume the judge is correct because it is consistent
- Why should both human and judge labels use exactly the same category set before building a confusion matrix?
- Because the LLM API requires identical categories to run at all
- Because it reduces the cost of running the judge
- Because kappa cannot be computed for text data
- Because mismatched categories make direct cross-tabulation invalid
Keep testing yourself
More quizzes picked for your level.