Guide-linked quiz · Hard Level
Inter-Annotator Agreement and Cohen's Kappa Quiz
Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.
Question 1 of 8
Why can raw percent agreement between two annotators be misleading, according to the guide?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- Why can raw percent agreement between two annotators be misleading, according to the guide?
- Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
- Because percent agreement is always mathematically identical to kappa
- Because percent agreement can never be calculated for binary labels
- Because it requires three or more annotators to compute
- What does Cohen's kappa specifically correct for that raw percent agreement does not?
- Annotator typing speed
- The expected agreement that would occur from chance given each annotator's individual label frequencies
- The total number of items labeled
- Whether annotators used the same software
- Which statistic is a common alternative for nominal labels when more than two raters are involved?
- Unweighted mean absolute error
- Fleiss’ kappa
- Pearson correlation only
- A weighted Cohen kappa that averages all coder pairs
- When should a weighted kappa variant be used instead of standard kappa?
- Only when there are more than 1000 labeled items
- Only for binary yes/no labels
- Never; weighted kappa is purely theoretical and not used in practice
- When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
- What value does Cohen’s kappa take when observed agreement equals the chance agreement estimated from the raters’ marginals?
- 0
- 1
- Negative infinity
- 100
- Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
- Raw agreement is not a valid calculation
- The marginal label frequencies can raise expected chance agreement
- Kappa ignores observed agreement
- A dominant class automatically means the labels are wrong
- Which statement about Krippendorff’s alpha is supported by the guide?
- A stricter, less flexible version of Cohen's kappa usable only for two raters
- A synonym for percent agreement with no chance correction
- A related alternative to kappa that handles missing data and more than two raters natively
- A measure that only applies to continuous numerical data, never categorical labels
- How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
- As a strict, universally agreed statistical standard set by an international body
- As values fixed by law in data science regulations
- As irrelevant, since kappa has no standard interpretation guidance at all
- As a widely used but informal scale, with thresholds varying by field
Keep testing yourself
More quizzes picked for your level.