Kvíz spojený s průvodcem · Těžko úroveň

Inter-Annotator Agreement and Cohen's Kappa Quiz

Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.

Otázka 1 z 8

Proč může být hrubá procentuální shoda mezi dvěma anotátory podle průvodce zavádějící?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. Proč může být hrubá procentuální shoda mezi dvěma anotátory podle průvodce zavádějící?
    • Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
    • Because percent agreement is always mathematically identical to kappa
    • Because percent agreement can never be calculated for binary labels
    • Because it requires three or more annotators to compute
  2. Co konkrétně Cohenův kappa opravuje, že hrubá procentuální shoda ne?
    • Annotator typing speed
    • The expected agreement that would occur from chance given each annotator's individual label frequencies
    • The total number of items labeled
    • Whether annotators used the same software
  3. Která statistika je běžnou alternativou pro nominální štítky, když jsou zapojeni více než dva hodnotitelé?
    • Unweighted mean absolute error
    • Fleiss’ kappa
    • Pearson correlation only
    • A weighted Cohen kappa that averages all coder pairs
  4. Kdy by měla být použita vážená varianta kappa místo standardní kappa?
    • Only when there are more than 1000 labeled items
    • Only for binary yes/no labels
    • Never; weighted kappa is purely theoretical and not used in practice
    • When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
  5. Jakou hodnotu má Cohenův kappa, když se pozorovaná shoda rovná náhodné shodě odhadované z marginálních hodnot hodnotitelů?
    • 0
    • 1
    • Negativní nekonečno
    • 100
  6. Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
    • Raw agreement is not a valid calculation
    • The marginal label frequencies can raise expected chance agreement
    • Kappa ignores observed agreement
    • A dominant class automatically means the labels are wrong
  7. Which statement about Krippendorff’s alpha is supported by the guide?
    • A stricter, less flexible version of Cohen's kappa usable only for two raters
    • A synonym for percent agreement with no chance correction
    • A related alternative to kappa that handles missing data and more than two raters natively
    • A measure that only applies to continuous numerical data, never categorical labels
  8. How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
    • As a strict, universally agreed statistical standard set by an international body
    • As values fixed by law in data science regulations
    • As irrelevant, since kappa has no standard interpretation guidance at all
    • As a widely used but informal scale, with thresholds varying by field