Next-Token Prediction Quiz

Answers and explanations appear as you take the quiz.

1. What does a next-token prediction model output at each step?
   • A probability distribution over the vocabulary for the next token
   • A single fixed word
   • A translation of the sentence
   • A grammar score
2. Which type of attention does a GPT-style decoder use?
   • Bidirectional attention
   • Causal (masked) attention that hides future tokens
   • No attention at all
   • Random attention
3. What does 'autoregressive' generation mean here?
   • The model generates all tokens at once
   • The model regresses to a mean value
   • Each new token is appended and fed back in to predict the next
   • Tokens are generated in random order
4. What loss function is typically minimized during training?
   • Mean squared error
   • Hinge loss
   • Cosine distance
   • Cross-entropy over the next token
5. What does raising the temperature during sampling do?
   • Makes output more random and creative
   • Makes output strictly deterministic
   • Increases the vocabulary size
   • Removes the softmax layer
6. Before prediction, how is raw text usually prepared?
   • It is translated to English
   • It is split into tokens by a tokenizer
   • It is converted to audio
   • It is left as raw characters with no processing
7. During next-token training, what is teacher forcing?
   • The model receives future target tokens before the input
   • The model randomly changes the tokenizer at every step
   • The model receives the ground-truth preceding tokens when predicting the next token
   • The model always samples its own previous prediction during training
8. What mismatch can arise because generation feeds back model-produced tokens?
   • The model can no longer use attention
   • The vocabulary disappears after the first token
   • The model’s loss is computed only on images
   • Errors in generated prefixes can affect later predictions, unlike training on the ground-truth prefix