Викторина, связанная с руководством · Жесткий Уровень
Калибровка судей LLM против людей. Викторина
Калибруйте судей LLM с помощью общих критериев, независимых человеческих ярлыков, матриц путаницы, подходящей статистики соглашений и повторяющихся отложенных проверок.
Вопрос 1 из 8
Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
Предварительный просмотр каждого вопроса8 вопросы
Ответы и пояснения появляются по мере прохождения теста.
- Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
- Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
- Kappa is always higher than percent agreement
- Kappa does not require human-labeled data
- Kappa can only be used with three or more raters
- What does a confusion matrix show when calibrating an LLM judge against human labels?
- The judge's response latency for each example
- How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
- The total token cost of running the judge
- The order in which examples were labeled
- In the example where a judge rates borderline-acceptable answers as 'excellent,' what does this pattern usually indicate?
- The judge model is fundamentally incapable of grading
- A hardware failure in the evaluation pipeline
- A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
- That human raters made a labeling error
- What can one round of judge calibration fail to account for?
- The model’s parameter count
- The number of UI buttons
- Whether the output includes a period
- The fact that a rubric or model version can change over time
- What does high agreement between an LLM judge and human ratings fail to prove by itself?
- That the rubric is valid or that the labels are correct
- As proof all human labels are correct
- As a reason to skip adjudication
- As a universal threshold for every task
- Why is measuring inter-human agreement useful before evaluating a judge?
- It has no real use in the calibration process
- As a useful reference point, not a universal cap or proof of correctness
- It replaces the need for a confusion matrix
- It determines the judge's token budget
- According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
- Hide the disagreements in an aggregate score
- Delete the category from every dataset
- Inspect those examples, clarify the category rubric, and retest on held-out cases
- Assume the judge is correct because it is consistent
- Why should both human and judge labels use exactly the same category set before building a confusion matrix?
- Because the LLM API requires identical categories to run at all
- Because it reduces the cost of running the judge
- Because kappa cannot be computed for text data
- Because mismatched categories make direct cross-tabulation invalid