Guide-linked quiz · Hard Level

Calibrating LLM Judges Against Humans Quiz

Calibrate LLM judges with shared rubrics, independent human labels, confusion matrices, suitable agreement statistics, and repeated held-out checks.

Question 1 of 8

Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
    • Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
    • Kappa is always higher than percent agreement
    • Kappa does not require human-labeled data
    • Kappa can only be used with three or more raters
  2. What does a confusion matrix show when calibrating an LLM judge against human labels?
    • The judge's response latency for each example
    • How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
    • The total token cost of running the judge
    • The order in which examples were labeled
  3. In the example where a judge rates borderline-acceptable answers as 'excellent,' what does this pattern usually indicate?
    • The judge model is fundamentally incapable of grading
    • A hardware failure in the evaluation pipeline
    • A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
    • That human raters made a labeling error
  4. What can one round of judge calibration fail to account for?
    • The model’s parameter count
    • The number of UI buttons
    • Whether the output includes a period
    • The fact that a rubric or model version can change over time
  5. What does high agreement between an LLM judge and human ratings fail to prove by itself?
    • That the rubric is valid or that the labels are correct
    • As proof all human labels are correct
    • As a reason to skip adjudication
    • As a universal threshold for every task
  6. Why is measuring inter-human agreement useful before evaluating a judge?
    • It has no real use in the calibration process
    • As a useful reference point, not a universal cap or proof of correctness
    • It replaces the need for a confusion matrix
    • It determines the judge's token budget
  7. According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
    • Hide the disagreements in an aggregate score
    • Delete the category from every dataset
    • Inspect those examples, clarify the category rubric, and retest on held-out cases
    • Assume the judge is correct because it is consistent
  8. Why should both human and judge labels use exactly the same category set before building a confusion matrix?
    • Because the LLM API requires identical categories to run at all
    • Because it reduces the cost of running the judge
    • Because kappa cannot be computed for text data
    • Because mismatched categories make direct cross-tabulation invalid

Keep testing yourself

More quizzes picked for your level.

View all quizzes