Cohen's Kappa Quiz

Answers and explanations appear as you take the quiz.

1. Two annotators have observed agreement of 0.8 and expected agreement of 0.5. Which kappa value follows?
   • 0.3
   • 0.4
   • 0.8
   • 0.6
2. One annotator uses a category 60% of the time and the other uses it 50% of the time. How much does that category contribute to expected agreement?
   • 0.10
   • 0.30
   • 0.55
   • 1.10
3. Why does the expected-agreement term in kappa not establish that annotators were guessing randomly?
   • It is calculated from the number of disagreements alone
   • It is identical to the observed agreement for every dataset
   • It is a reference derived from label frequencies, not an observation of their reasoning
   • It assumes every category must have equal frequency
4. For low, medium and high incident severity, why might a team choose weighted kappa?
   • To give different penalties to one-level and two-level disagreements
   • To remove all disagreements from the calculation
   • To guarantee a higher reliability score
   • To avoid specifying a category order
5. Both annotators assign every case to the same single category. Why should ordinary kappa not be reported as a straightforward reliability success?
   • Perfect raw agreement automatically makes kappa one
   • The score ignores the common category and evaluates only disagreements
   • Subtracting expected agreement alone gives a complete kappa calculation
   • Expected agreement is one, leaving a zero denominator
6. A second reviewer copies the first reviewer's labels and achieves perfect agreement. Which conclusion is justified?
   • The labels have been independently validated
   • The agreement does not establish independent reliability
   • Every label is necessarily correct
   • The dataset can be used without checking instructions
7. Which accompanying information helps interpret a kappa score when category frequencies are uneven?
   • Only the raw agreement percentage, without category counts
   • Only whether kappa exceeds a universal cutoff
   • Sample size, raw agreement and the agreement table
   • Only the weighted score, without specifying category order
8. Two annotators both follow a mistaken rule and agree strongly. What does their high kappa fail to establish?
   • That their labels are factually correct
   • That their observed agreement exceeds its frequency-based expectation
   • That they used the same labels on many cases
   • That their assignments show consistency under the shared rule