Ikibazo gihuza ibibazo · Biragoye Urwego

Guhindura Abacamanza ba LLM Kurwanya Abantu Ikibazo

Hindura abacamanza ba LLM hamwe na rubrics basangiye, ibirango byigenga byabantu, matriche yo kwitiranya, imibare ikwiye yamasezerano, hamwe na cheque yagenzuwe kenshi.

Ikibazo 1 Bya 8

Kuki kappa ya Cohen ikundwa kuruta amasezerano yijana mugihe uhindura umucamanza wa LLM?

Reba buri kibazo8 ibibazo

Ibisubizo nibisobanuro bigaragara mugihe ufata ikibazo.

  1. Kuki kappa ya Cohen ikundwa kuruta amasezerano yijana mugihe uhindura umucamanza wa LLM?
    • Kappa corrects for the agreement expected by pure chance, which raw percent agreement can overstate
    • Kappa is always higher than percent agreement
    • Kappa does not require human-labeled data
    • Kappa can only be used with three or more raters
  2. Matrix yo kwitiranya yerekana iki mugihe uhindura umucamanza wa LLM kurwanya ibirango byabantu?
    • The judge's response latency for each example
    • How human labels and judge labels cross-tabulate, revealing exactly where disagreement is concentrated
    • The total token cost of running the judge
    • The order in which examples were labeled
  3. Murugero aho umucamanza agereranya ibisubizo byemewe kumipaka nk '' byiza, 'ubu buryo bukunze kwerekana iki?
    • The judge model is fundamentally incapable of grading
    • A hardware failure in the evaluation pipeline
    • A category-specific mismatch to investigate across rubric clarity, judge behavior, and human disagreement
    • That human raters made a labeling error
  4. Niki cyiciro kimwe cyabacamanza cyananiwe kubara?
    • Icyitegererezo cyo kubara icyitegererezo
    • The number of UI buttons
    • Whether the output includes a period
    • The fact that a rubric or model version can change over time
  5. Ni ayahe masezerano menshi hagati yumucamanza wa LLM nu manota yabantu yananiwe kwerekana wenyine?
    • That the rubric is valid or that the labels are correct
    • As proof all human labels are correct
    • As a reason to skip adjudication
    • As a universal threshold for every task
  6. Why is measuring inter-human agreement useful before evaluating a judge?
    • It has no real use in the calibration process
    • As a useful reference point, not a universal cap or proof of correctness
    • It replaces the need for a confusion matrix
    • It determines the judge's token budget
  7. According to the guide, what is a practical response when a judge repeatedly disagrees with humans on one category?
    • Hide the disagreements in an aggregate score
    • Delete the category from every dataset
    • Inspect those examples, clarify the category rubric, and retest on held-out cases
    • Assume the judge is correct because it is consistent
  8. Why should both human and judge labels use exactly the same category set before building a confusion matrix?
    • Because the LLM API requires identical categories to run at all
    • Because it reduces the cost of running the judge
    • Because kappa cannot be computed for text data
    • Because mismatched categories make direct cross-tabulation invalid