Guide-linked quiz · Hard Level

Inter-Annotator Agreement and Cohen's Kappa Quiz

Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.

Question 1 of 8

Why can raw percent agreement between two annotators be misleading, according to the guide?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Why can raw percent agreement between two annotators be misleading, according to the guide?
    • Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
    • Because percent agreement is always mathematically identical to kappa
    • Because percent agreement can never be calculated for binary labels
    • Because it requires three or more annotators to compute
  2. What does Cohen's kappa specifically correct for that raw percent agreement does not?
    • Annotator typing speed
    • The expected agreement that would occur from chance given each annotator's individual label frequencies
    • The total number of items labeled
    • Whether annotators used the same software
  3. Which statistic is a common alternative for nominal labels when more than two raters are involved?
    • Unweighted mean absolute error
    • Fleiss’ kappa
    • Pearson correlation only
    • A weighted Cohen kappa that averages all coder pairs
  4. When should a weighted kappa variant be used instead of standard kappa?
    • Only when there are more than 1000 labeled items
    • Only for binary yes/no labels
    • Never; weighted kappa is purely theoretical and not used in practice
    • When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
  5. What value does Cohen’s kappa take when observed agreement equals the chance agreement estimated from the raters’ marginals?
    • 0
    • 1
    • Negative infinity
    • 100
  6. Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
    • Raw agreement is not a valid calculation
    • The marginal label frequencies can raise expected chance agreement
    • Kappa ignores observed agreement
    • A dominant class automatically means the labels are wrong
  7. Which statement about Krippendorff’s alpha is supported by the guide?
    • A stricter, less flexible version of Cohen's kappa usable only for two raters
    • A synonym for percent agreement with no chance correction
    • A related alternative to kappa that handles missing data and more than two raters natively
    • A measure that only applies to continuous numerical data, never categorical labels
  8. How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
    • As a strict, universally agreed statistical standard set by an international body
    • As values fixed by law in data science regulations
    • As irrelevant, since kappa has no standard interpretation guidance at all
    • As a widely used but informal scale, with thresholds varying by field

Keep testing yourself

More quizzes picked for your level.

View all quizzes