Kvíz spojený s průvodcem · Střední úroveň

Label Bias and Annotator Disagreement Quiz

Examine subjective labels, annotator identity and instructions, proxy targets, disagreement, and safer ways to preserve uncertainty in training data.

Související vodicí cestyLabel Bias And Annotator Disagreement
Otázka 1 z 8

A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. A model learns “toxic” labels that reflect annotators’ assumptions about dialect. What risk is most direct?
    • Label bias
    • Sampling latency
    • Hardware failure
    • Komprese obrazu
  2. What did Sap and colleagues find in research on toxicity labels and African American English?
    • Dialect markers were unrelated to annotation
    • Annotator identity and beliefs were associated with toxicity ratings in their studies
    • All annotators agreed on every post
    • The model never used the training labels
  3. Why may disagreement on sarcasm or offensive language be meaningful?
    • It always proves the dataset is corrupt
    • It only occurs with missing data
    • The task can be subjective and people may interpret context differently
    • A model cannot accept labels
  4. What can happen when a team turns every disagreement into one majority label?
    • Minority perspectives and uncertainty may be erased
    • The model automatically becomes more fair
    • The dataset becomes representative
    • The labels become legally binding
  5. Why is health-care spending a problematic proxy for medical need?
    • Spending is always random
    • Past spending can reflect access barriers and unequal care
    • Costs are never recorded
    • Models cannot use numerical values
  6. Which practice helps audit label provenance?
    • Store only the final majority label
    • Delete annotator instructions
    • Preserve individual annotations and instructions before aggregation
    • Replace labels with model predictions
  7. When may a majority label be insufficient?
    • When the application needs to represent ambiguity or multiple perspectives
    • When the dataset has a file name
    • When every judgment agrees
    • When the model is deployed offline
  8. How should teams interpret low annotator agreement?
    • As automatic proof of bad annotators
    • As a possible sign of a subjective or underspecified task that needs investigation
    • As proof the model is accurate
    • As a reason to hide the labels