Kvíz spojený s průvodcem · Těžko úroveň

Calibrating LLM Judges Against Humans Quiz

Calibrate LLM judges with shared rubrics, independent human labels, confusion matrices, suitable agreement statistics, and repeated held-out checks.

Související vodicí cestyKalibrace LLM soudců vůči lidem
Otázka 1 z 8

Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
    • Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
    • Kappa is always higher than percent agreement
    • Kappa does not require human-labeled data
    • Kappa can only be used with three or more raters
  2. What does a confusion matrix show when calibrating an LLM judge against human labels?
    • The judge's response latency for each example
    • How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
    • The total token cost of running the judge
    • The order in which examples were labeled
  3. In the example where a judge rates borderline-acceptable answers as 'excellent,' what does this pattern usually indicate?
    • The judge model is fundamentally incapable of grading
    • A hardware failure in the evaluation pipeline
    • A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
    • That human raters made a labeling error
  4. What can one round of judge calibration fail to account for?
    • The model’s parameter count
    • The number of UI buttons
    • Whether the output includes a period
    • The fact that a rubric or model version can change over time
  5. What does high agreement between an LLM judge and human ratings fail to prove by itself?
    • That the rubric is valid or that the labels are correct
    • As proof all human labels are correct
    • As a reason to skip adjudication
    • As a universal threshold for every task
  6. Why is measuring inter-human agreement useful before evaluating a judge?
    • It has no real use in the calibration process
    • As a useful reference point, not a universal cap or proof of correctness
    • It replaces the need for a confusion matrix
    • It determines the judge's token budget
  7. According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
    • Hide the disagreements in an aggregate score
    • Delete the category from every dataset
    • Inspect those examples, clarify the category rubric, and retest on held-out cases
    • Assume the judge is correct because it is consistent
  8. Why should both human and judge labels use exactly the same category set before building a confusion matrix?
    • Because the LLM API requires identical categories to run at all
    • Because it reduces the cost of running the judge
    • Because kappa cannot be computed for text data
    • Because mismatched categories make direct cross-tabulation invalid