Guide-yakabatana mibvunzo · Zvakaoma Nhanho

Inter-Annotator Agreement and Cohen's Kappa Quiz

Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.

Nzira dzekutungamira dzinoenderanaInter Annotator Agreement And Cohens Kappa
Mubvunzo 1 ye 8

Why can raw percent agreement between two annotators be misleading, according to the guide?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Why can raw percent agreement between two annotators be misleading, according to the guide?
    • Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
    • Because percent agreement is always mathematically identical to kappa
    • Because percent agreement can never be calculated for binary labels
    • Because it requires three or more annotators to compute
  2. What does Cohen's kappa specifically correct for that raw percent agreement does not?
    • Annotator typing speed
    • The expected agreement that would occur from chance given each annotator's individual label frequencies
    • The total number of items labeled
    • Whether annotators used the same software
  3. Ndeipi nhamba inowanzo shandiswa sechinhu chinotsiva mazita epamusoro kana pane vanoverenga vakawanda kupfuura maviri?
    • Unweighted mean absolute error
    • Fleiss’ kappa
    • Pearson correlation only
    • A weighted Cohen kappa that averages all coder pairs
  4. Pachii panofanira kushandiswa kappa yakarongedzwa pane kappa yakajairika?
    • Only when there are more than 1000 labeled items
    • Only for binary yes/no labels
    • Never; weighted kappa is purely theoretical and not used in practice
    • When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
  5. Chii chinowanikwa neCohen’s kappa kana kubvumirana kwacho kuchienderana ne chance agreement yakaverengwa kubva kumarginals evanoverenga?
    • 0
    • 1
    • Negative infinity
    • 100
  6. Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
    • Raw agreement is not a valid calculation
    • The marginal label frequencies can raise expected chance agreement
    • Kappa ignores observed agreement
    • A dominant class automatically means the labels are wrong
  7. Which statement about Krippendorff’s alpha is supported by the guide?
    • A stricter, less flexible version of Cohen's kappa usable only for two raters
    • A synonym for percent agreement with no chance correction
    • A related alternative to kappa that handles missing data and more than two raters natively
    • A measure that only applies to continuous numerical data, never categorical labels
  8. How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
    • As a strict, universally agreed statistical standard set by an international body
    • As values fixed by law in data science regulations
    • As irrelevant, since kappa has no standard interpretation guidance at all
    • As a widely used but informal scale, with thresholds varying by field