GUIDE Technique
Cohen's Kappa
Cohen's kappa measures agreement between two sets of categorical labels after accounting for agreement expected from their label frequencies.
Sur cette page3 minutes de lecture
Aperçu
It helps teams assess annotation consistency, but a high score does not establish that either annotator is factually correct.
Plongée profonde
Suppose two annotators agree on most examples. That sounds reassuring, but the category frequencies matter. If both almost always choose the same common category, frequent agreement can arise without much ability to distinguish cases. Cohen's kappa compares observed agreement with an expected-agreement calculation based on each annotator's label proportions. The formula subtracts expected agreement from observed agreement, then divides by one minus expected agreement. If observed agreement is 0.8 and expected agreement is 0.5, kappa is 0.6. These values form an illustrative calculation. A kappa of one represents perfect agreement when the denominator is defined; zero means agreement matches the frequency-based expectation. Negative values indicate less agreement than that expectation. The expected term does not prove that either person guessed randomly. It is a mathematical reference constructed from the marginal label frequencies. For example, if one annotator uses a category 60% of the time and the other uses it 50% of the time, that category contributes 0.3 to expected agreement. Add the contributions for every category. Use independently assigned labels on the same cases. If the second annotator copies the first, their agreement says little about independent reliability. Examine disagreement patterns before changing the instructions, then evaluate revised instructions on a fresh sample where practical. Kappa also depends on category prevalence and the annotators' use of labels. There is no universally appropriate cutoff that makes a dataset trustworthy. Report sample size, raw agreement and the agreement table alongside the score. Scikit-learn offers unweighted and weighted calculations. Weighted kappa is suitable when category order and the consequences of different-sized disagreements have been defined clearly.
Impact stratégique
Coût et budget
Les décisions en matière d'architecture déterminent les performances et les coûts d'exploitation pendant des années.
Décisions plus claires
La formation technique aide les équipes à choisir la bonne pile, pas seulement la plus récente.
Contrôle qualité
De meilleurs choix d’ingénierie réduisent les incidents de fiabilité en production.
The Future of Cohen's Kappa
AI-assisted annotation makes it more important to record how each label was produced. Two reviewers who saw the same model suggestion may share an error, even when their agreement is high. Teams can strengthen their process by preserving a sample labeled independently, keeping a record of instruction changes and discussing recurring disagreements with domain experts. Kappa can remain one part of that review, alongside checks against well-supported reference answers. The goal is a dataset whose labels have a clear meaning and a defensible review process, rather than a single impressive agreement number.
Mise en œuvre dans le monde réel
Two people independently label the same support tickets as billing or technical. A kappa report measures their agreement and a disagreement review reveals where the labeling instructions need clarification.
In a hypothetical sample, observed agreement is 80% and expected agreement is 50%. Kappa is (0.8 minus 0.5) divided by (1 minus 0.5), or 0.6.
Reviewers assign low, medium or high severity to incidents. A weighted kappa can distinguish a one-level disagreement from a disagreement between the lowest and highest levels.
An analyst uses scikit-learn's cohen_kappa_score for two aligned label arrays, then reports the category counts and agreement table so readers can interpret the summary.
Risques et garde-fous
L’optimisation d’un benchmark peut masquer des faiblesses plus larges du système.
Les coûts d’infrastructure et de maintenance sont souvent sous-estimés.
Les lacunes en matière de sécurité et d’observabilité peuvent se creuser à mesure que les systèmes deviennent plus complexes.
Feuille de route de mise en œuvre
Définissez les objectifs de latence, de qualité et de coût avant la mise en œuvre.
Benchmark dans des conditions de charge et de données réalistes.
Surveillance des instruments pour détecter les erreurs, la dérive et l'impact sur l'utilisateur.
Préparez les chemins de restauration et de réponse aux incidents avant la mise à l’échelle.
Continuez à explorer
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Cohen's Kappa quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Questions fréquemment posées
What is Cohen's Kappa?
Cohen's kappa measures agreement between two sets of categorical labels after accounting for agreement expected from their label frequencies. It helps teams assess annotation consistency, but a high score does not establish that either annotator is factually correct.
Two annotators have observed agreement of 0.8 and expected agreement of 0.5. Which kappa value follows?
Kappa is (0.8 minus 0.5) divided by (1 minus 0.5), which equals 0.6.
Why does the expected-agreement term in kappa not establish that annotators were guessing randomly?
The expected term is a mathematical baseline constructed from marginal frequencies; it does not describe the annotators' actual thought process.
For low, medium and high incident severity, why might a team choose weighted kappa?
Weights allow the score to reflect the distance between ordered categories, provided that order and weighting are appropriate.
Both annotators assign every case to the same single category. Why should ordinary kappa not be reported as a straightforward reliability success?
The denominator is one minus expected agreement. When expected agreement is one, the ordinary formula is undefined.
A second reviewer copies the first reviewer's labels and achieves perfect agreement. Which conclusion is justified?
Copied decisions do not provide an independent assessment, even if their numerical agreement is perfect.
Continuez à apprendre
Guides associés
Plus de guides sélectionnés pour ce sujet