Викторина, связанная с руководством · Жесткий Уровень

Соглашение между аннотаторами и каппа-викторина Коэна

Проверяет понимание формулы каппы Коэна, поправки на случайность, ограничения на двух оценщиков и связанных с ней статистических данных, таких как каппа Флейса.

Вопрос 1 из 8

Why can raw percent agreement between two annotators be misleading, according to the guide?

Предварительный просмотр каждого вопроса8 вопросы

Ответы и пояснения появляются по мере прохождения теста.

  1. Why can raw percent agreement between two annotators be misleading, according to the guide?
    • Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
    • Because percent agreement is always mathematically identical to kappa
    • Because percent agreement can never be calculated for binary labels
    • Because it requires three or more annotators to compute
  2. What does Cohen's kappa specifically correct for that raw percent agreement does not?
    • Annotator typing speed
    • The expected agreement that would occur from chance given each annotator's individual label frequencies
    • The total number of items labeled
    • Whether annotators used the same software
  3. Which statistic is a common alternative for nominal labels when more than two raters are involved?
    • Unweighted mean absolute error
    • Fleiss’ kappa
    • Pearson correlation only
    • A weighted Cohen kappa that averages all coder pairs
  4. When should a weighted kappa variant be used instead of standard kappa?
    • Only when there are more than 1000 labeled items
    • Only for binary yes/no labels
    • Never; weighted kappa is purely theoretical and not used in practice
    • When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
  5. What value does Cohen’s kappa take when observed agreement equals the chance agreement estimated from the raters’ marginals?
    • 0
    • 1
    • Отрицательная бесконечность
    • 100
  6. Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
    • Raw agreement is not a valid calculation
    • The marginal label frequencies can raise expected chance agreement
    • Kappa ignores observed agreement
    • A dominant class automatically means the labels are wrong
  7. Which statement about Krippendorff’s alpha is supported by the guide?
    • A stricter, less flexible version of Cohen's kappa usable only for two raters
    • A synonym for percent agreement with no chance correction
    • A related alternative to kappa that handles missing data and more than two raters natively
    • A measure that only applies to continuous numerical data, never categorical labels
  8. How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
    • As a strict, universally agreed statistical standard set by an international body
    • As values fixed by law in data science regulations
    • As irrelevant, since kappa has no standard interpretation guidance at all
    • As a widely used but informal scale, with thresholds varying by field