Masked Language Modeling Quiz

Answers and explanations appear as you take the quiz.

1. What is the core training task in masked language modeling?
   • Predicting deliberately hidden words from surrounding context
   • Predicting the next word after a sentence
   • Translating text between two languages
   • Ranking sentences by grammatical correctness
2. Roughly what percentage of tokens does BERT typically mask during pretraining?
   • About 50%
   • About 15%
   • About 1%
   • About 90%
3. Why does MLM give a model bidirectional understanding?
   • It reads sentences backward
   • It uses two separate models
   • Each token attends to words on both its left and right
   • It only looks at the previous word
4. In BERT's masking scheme, what happens to about 10% of selected positions instead of becoming [MASK]?
   • They are deleted entirely
   • They are duplicated
   • They are capitalized
   • They are replaced with a random word
5. On which positions is the MLM loss computed?
   • Only the masked positions
   • Every token in the sentence
   • Only the first and last tokens
   • Only punctuation tokens
6. Compared with next-token prediction, what does MLM trade away?
   • The ability to use context
   • The ability to generate fluent text left-to-right
   • Access to a vocabulary
   • The use of attention
7. How does RoBERTa’s dynamic masking differ from BERT’s original setup?
   • It masks only the first token of every sentence
   • It predicts future tokens with a causal decoder
   • The masked positions can be regenerated across training passes instead of remaining fixed
   • It replaces all masked tokens with random numbers
8. What auxiliary objective did the original BERT pretraining recipe combine with MLM?
   • Image patch reconstruction
   • Audio speaker diarization
   • A reward model for code generation
   • Next-sentence prediction