Bài kiểm tra có liên kết hướng dẫn · Cứng Cấp độ

Thỏa thuận giữa các nhà chú thích và Câu hỏi Kappa của Cohen

Kiểm tra sự hiểu biết về công thức kappa của Cohen, hiệu chỉnh cơ hội, giới hạn cho hai người xếp hạng và các số liệu thống kê liên quan như kappa của Fleiss.

Đường dẫn hướng dẫn liên quanThỏa thuận Inter Annotator và Cohens Kappa
Câu hỏi 1 của 8

Why can raw percent agreement between two annotators be misleading, according to the guide?

Xem trước mọi câu hỏi8 câu hỏi

Câu trả lời và giải thích xuất hiện khi bạn làm bài kiểm tra.

  1. Why can raw percent agreement between two annotators be misleading, according to the guide?
    • Because it doesn't account for agreement that could occur just by chance, especially with skewed label distributions
    • Because percent agreement is always mathematically identical to kappa
    • Because percent agreement can never be calculated for binary labels
    • Because it requires three or more annotators to compute
  2. What does Cohen's kappa specifically correct for that raw percent agreement does not?
    • Annotator typing speed
    • The expected agreement that would occur from chance given each annotator's individual label frequencies
    • The total number of items labeled
    • Whether annotators used the same software
  3. Which statistic is a common alternative for nominal labels when more than two raters are involved?
    • Unweighted mean absolute error
    • Fleiss’ kappa
    • Pearson correlation only
    • A weighted Cohen kappa that averages all coder pairs
  4. When should a weighted kappa variant be used instead of standard kappa?
    • Only when there are more than 1000 labeled items
    • Only for binary yes/no labels
    • Never; weighted kappa is purely theoretical and not used in practice
    • When labels have a natural order, so a one-point disagreement should count less than a larger disagreement
  5. What value does Cohen’s kappa take when observed agreement equals the chance agreement estimated from the raters’ marginals?
    • 0
    • 1
    • âm vô cùng
    • 100
  6. Why can kappa be low even when raw agreement looks high in a dataset with a dominant class?
    • Raw agreement is not a valid calculation
    • The marginal label frequencies can raise expected chance agreement
    • Kappa ignores observed agreement
    • A dominant class automatically means the labels are wrong
  7. Which statement about Krippendorff’s alpha is supported by the guide?
    • A stricter, less flexible version of Cohen's kappa usable only for two raters
    • A synonym for percent agreement with no chance correction
    • A related alternative to kappa that handles missing data and more than two raters natively
    • A measure that only applies to continuous numerical data, never categorical labels
  8. How are the commonly cited interpretation bands for kappa values (e.g., 'slight,' 'almost perfect') described in the guide?
    • As a strict, universally agreed statistical standard set by an international body
    • As values fixed by law in data science regulations
    • As irrelevant, since kappa has no standard interpretation guidance at all
    • As a widely used but informal scale, with thresholds varying by field