แบบทดสอบที่เชื่อมโยงกับคำแนะนำ · ยาก ระดับ
แบบทดสอบการสอบเทียบผู้พิพากษา LLM กับมนุษย์
ปรับเทียบผู้ตัดสิน LLM ด้วยรูบริกที่ใช้ร่วมกัน ป้ายกำกับที่เป็นอิสระจากมนุษย์ เมทริกซ์ความสับสน สถิติข้อตกลงที่เหมาะสม และการตรวจสอบที่ค้างอยู่ซ้ำแล้วซ้ำอีก
คำถาม 1 ของ 8
Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
ดูตัวอย่างทุกคำถาม8 คำถาม
คำตอบและคำอธิบายจะปรากฏขึ้นเมื่อคุณทำแบบทดสอบ
- Why is Cohen's kappa preferred over raw percent agreement when calibrating an LLM judge?
- Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
- Kappa is always higher than percent agreement
- Kappa does not require human-labeled data
- Kappa can only be used with three or more raters
- What does a confusion matrix show when calibrating an LLM judge against human labels?
- The judge's response latency for each example
- How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
- The total token cost of running the judge
- The order in which examples were labeled
- In the example where a judge rates borderline-acceptable answers as 'excellent,' what does this pattern usually indicate?
- The judge model is fundamentally incapable of grading
- A hardware failure in the evaluation pipeline
- A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
- That human raters made a labeling error
- What can one round of judge calibration fail to account for?
- จำนวนพารามิเตอร์ของโมเดล
- The number of UI buttons
- Whether the output includes a period
- The fact that a rubric or model version can change over time
- What does high agreement between an LLM judge and human ratings fail to prove by itself?
- That the rubric is valid or that the labels are correct
- As proof all human labels are correct
- As a reason to skip adjudication
- As a universal threshold for every task
- Why is measuring inter-human agreement useful before evaluating a judge?
- It has no real use in the calibration process
- As a useful reference point, not a universal cap or proof of correctness
- It replaces the need for a confusion matrix
- It determines the judge's token budget
- According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
- Hide the disagreements in an aggregate score
- Delete the category from every dataset
- Inspect those examples, clarify the category rubric, and retest on held-out cases
- Assume the judge is correct because it is consistent
- Why should both human and judge labels use exactly the same category set before building a confusion matrix?
- Because the LLM API requires identical categories to run at all
- Because it reduces the cost of running the judge
- Because kappa cannot be computed for text data
- Because mismatched categories make direct cross-tabulation invalid