Kvíz spojený s průvodcem · Těžko úroveň
Inter-Annotator Agreement and Cohen's Kappa Quiz
Tests understanding of Cohen's kappa's formula, chance correction, limits to two raters, and related statistics like Fleiss' kappa.
Otázka 1 z 8
Proč může být hrubá procentuální shoda mezi dvěma anotátory podle průvodce zavádějící?
Náhled každé otázky8 otázky
Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.
- Proč může být hrubá procentuální shoda mezi dvěma anotátory podle průvodce zavádějící?
- Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
- Because percent agreement is always mathematically identical to kappa
- Because percent agreement can never be calculated for binary labels
- Because it requires three or more annotators to compute
- Co konkrétně Cohenův kappa opravuje, že hrubá procentuální shoda ne?
- Annotator typing speed
- The expected agreement that would occur from chance given each annotator's individual label frequencies
- The total number of items labeled
- Whether annotators used the same software
- Která statistika je běžnou alternativou pro nominální štítky, když jsou zapojeni více než dva hodnotitelé?
- Unweighted mean absolute error
- Fleiss’ kappa
- Pearson correlation only
- A weighted Cohen kappa that averages all coder pairs
- Kdy by měla být použita vážená varianta kappa místo standardní kappa?
- Only when there are more than 1000 labeled items
- Only for binary yes/no labels
- Never; weighted kappa is purely theoretical and not used in practice
- When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
- Jakou hodnotu má Cohenův kappa, když se pozorovaná shoda rovná náhodné shodě odhadované z marginálních hodnot hodnotitelů?
- 0
- 1
- Negativní nekonečno
- 100
- Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
- Raw agreement is not a valid calculation
- The marginal label frequencies can raise expected chance agreement
- Kappa ignores observed agreement
- A dominant class automatically means the labels are wrong
- Which statement about Krippendorff’s alpha is supported by the guide?
- A stricter, less flexible version of Cohen's kappa usable only for two raters
- A synonym for percent agreement with no chance correction
- A related alternative to kappa that handles missing data and more than two raters natively
- A measure that only applies to continuous numerical data, never categorical labels
- How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
- As a strict, universally agreed statistical standard set by an international body
- As values fixed by law in data science regulations
- As irrelevant, since kappa has no standard interpretation guidance at all
- As a widely used but informal scale, with thresholds varying by field