Guide-linked quiz · Medium Level

Label Bias and Annotator Disagreement Quiz

Examine subjective labels, annotator identity and instructions, proxy targets, disagreement, and safer ways to preserve uncertainty in training data.

Question 1 of 8

A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?
    • Label bias
    • Sampling latency
    • Hardware failure
    • Image compression
  2. What did Sap and colleagues find in research on toxicity labels and African American English?
    • Dialect markers were unrelated to annotation
    • Annotator identity and beliefs were associated with toxicity ratings in their studies
    • All annotators agreed on every post
    • The model never used the training labels
  3. Why may disagreement on sarcasm or offensive language be meaningful?
    • It always proves the dataset is corrupt
    • It only occurs with missing data
    • The task can be subjective and people may interpret context differently
    • A model cannot accept labels
  4. What can happen when a team turns every disagreement into one majority label?
    • Minority perspectives and uncertainty may be erased
    • The model automatically becomes more fair
    • The dataset becomes representative
    • The labels become legally binding
  5. Why is health-care spending a problematic proxy for medical need?
    • Spending is always random
    • Past spending can reflect access barriers and unequal care
    • Costs are never recorded
    • Models cannot use numerical values
  6. Which practice helps audit label provenance?
    • Store only the final majority label
    • Delete annotator instructions
    • Preserve individual annotations and instructions before aggregation
    • Replace labels with model predictions
  7. When may a majority label be insufficient?
    • When the application needs to represent ambiguity or multiple perspectives
    • When the dataset has a file name
    • When every judgment agrees
    • When the model is deployed offline
  8. How should teams interpret low annotator agreement?
    • As automatic proof of bad annotators
    • As a possible sign of a subjective or underspecified task that needs investigation
    • As proof the model is accurate
    • As a reason to hide the labels

Keep testing yourself

More quizzes picked for your level.

View all quizzes