Guide-yakabatana mibvunzo · Zvakaoma Nhanho
Inter-Annotator Agreement and Cohen's Kappa Quiz
Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.
Mubvunzo 1 ye 8
Why can raw percent agreement between two annotators be misleading, according to the guide?
Tarisa mubvunzo wese8 mibvunzo
Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.
- Why can raw percent agreement between two annotators be misleading, according to the guide?
- Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
- Because percent agreement is always mathematically identical to kappa
- Because percent agreement can never be calculated for binary labels
- Because it requires three or more annotators to compute
- What does Cohen's kappa specifically correct for that raw percent agreement does not?
- Annotator typing speed
- The expected agreement that would occur from chance given each annotator's individual label frequencies
- The total number of items labeled
- Whether annotators used the same software
- Ndeipi nhamba inowanzo shandiswa sechinhu chinotsiva mazita epamusoro kana pane vanoverenga vakawanda kupfuura maviri?
- Unweighted mean absolute error
- Fleiss’ kappa
- Pearson correlation only
- A weighted Cohen kappa that averages all coder pairs
- Pachii panofanira kushandiswa kappa yakarongedzwa pane kappa yakajairika?
- Only when there are more than 1000 labeled items
- Only for binary yes/no labels
- Never; weighted kappa is purely theoretical and not used in practice
- When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
- Chii chinowanikwa neCohen’s kappa kana kubvumirana kwacho kuchienderana ne chance agreement yakaverengwa kubva kumarginals evanoverenga?
- 0
- 1
- Negative infinity
- 100
- Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
- Raw agreement is not a valid calculation
- The marginal label frequencies can raise expected chance agreement
- Kappa ignores observed agreement
- A dominant class automatically means the labels are wrong
- Which statement about Krippendorff’s alpha is supported by the guide?
- A stricter, less flexible version of Cohen's kappa usable only for two raters
- A synonym for percent agreement with no chance correction
- A related alternative to kappa that handles missing data and more than two raters natively
- A measure that only applies to continuous numerical data, never categorical labels
- How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
- As a strict, universally agreed statistical standard set by an international body
- As values fixed by law in data science regulations
- As irrelevant, since kappa has no standard interpretation guidance at all
- As a widely used but informal scale, with thresholds varying by field