Guide-linked quiz · Medium Level
Label Bias and Annotator Disagreement Quiz
Examine subjective labels, annotator identity and instructions, proxy targets, disagreement, and safer ways to preserve uncertainty in training data.
Question 1 of 8
A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?
- Label bias
- Sampling latency
- Hardware failure
- Image compression
- What did Sap and colleagues find in research on toxicity labels and African American English?
- Dialect markers were unrelated to annotation
- Annotator identity and beliefs were associated with toxicity ratings in their studies
- All annotators agreed on every post
- The model never used the training labels
- Why may disagreement on sarcasm or offensive language be meaningful?
- It always proves the dataset is corrupt
- It only occurs with missing data
- The task can be subjective and people may interpret context differently
- A model cannot accept labels
- What can happen when a team turns every disagreement into one majority label?
- Minority perspectives and uncertainty may be erased
- The model automatically becomes more fair
- The dataset becomes representative
- The labels become legally binding
- Why is health-care spending a problematic proxy for medical need?
- Spending is always random
- Past spending can reflect access barriers and unequal care
- Costs are never recorded
- Models cannot use numerical values
- Which practice helps audit label provenance?
- Store only the final majority label
- Delete annotator instructions
- Preserve individual annotations and instructions before aggregation
- Replace labels with model predictions
- When may a majority label be insufficient?
- When the application needs to represent ambiguity or multiple perspectives
- When the dataset has a file name
- When every judgment agrees
- When the model is deployed offline
- How should teams interpret low annotator agreement?
- As automatic proof of bad annotators
- As a possible sign of a subjective or underspecified task that needs investigation
- As proof the model is accurate
- As a reason to hide the labels
Keep testing yourself
More quizzes picked for your level.