Cohen's Kappa

Suppose two annotators agree on most examples. That sounds reassuring, but the category frequencies matter. If both almost always choose the same common category, frequent agreement can arise without much ability to distinguish cases. Cohen's kappa compares observed agreement with an expected-agreement calculation based on each annotator's label proportions. The formula subtracts expected agreement from observed agreement, then divides by one minus expected agreement. If observed agreement is 0.8 and expected agreement is 0.5, kappa is 0.6. These values form an illustrative calculation. A kappa of one represents perfect agreement when the denominator is defined; zero means agreement matches the frequency-based expectation. Negative values indicate less agreement than that expectation. The expected term does not prove that either person guessed randomly. It is a mathematical reference constructed from the marginal label frequencies. For example, if one annotator uses a category 60% of the time and the other uses it 50% of the time, that category contributes 0.3 to expected agreement. Add the contributions for every category. Use independently assigned labels on the same cases. If the second annotator copies the first, their agreement says little about independent reliability. Examine disagreement patterns before changing the instructions, then evaluate revised instructions on a fresh sample where practical. Kappa also depends on category prevalence and the annotators' use of labels. There is no universally appropriate cutoff that makes a dataset trustworthy. Report sample size, raw agreement and the agreement table alongside the score. Scikit-learn offers unweighted and weighted calculations. Weighted kappa is suitable when category order and the consequences of different-sized disagreements have been defined clearly.