Guide-linked quiz · Medium Level
Inverse Scaling in Language Models Quiz
Interpret inverse scaling with task scores, model-series controls, memorized patterns, distractors, uncertainty, and possible trend reversals.
Question 1 of 8
Three increasingly large models in a comparable series score 80%, 70%, and 60% on the same accuracy task. What does this show?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- Three increasingly large models in a comparable series score 80%, 70%, and 60% on the same accuracy task. What does this show?
- An inverse trend on that task over the tested sizes
- That every capability declines with model size
- That all three models have identical training loss
- That a larger fourth model must score below 60%
- Why can better average next-token loss coexist with worse performance on an instruction-following task?
- The two measures always evaluate the same objective
- Predicting familiar text and following a changed instruction can favor different outputs
- Lower next-token loss always means worse language modeling
- Task accuracy is independent of the model’s output
- A prompt supplies an unusual ending to a familiar phrase, but the model uses the familiar ending. Which proposed failure pattern does this illustrate?
- An increase in accelerator memory
- A reduction in vocabulary size
- A memorized continuation taking precedence over the supplied instruction
- A guaranteed improvement in task accuracy
- A fourth, larger model scores 85% after the sequence 80%, 70%, and 60%. How should the evaluation report this?
- Discard the fourth result to preserve the original claim
- Declare that the earlier measurements never happened
- Predict all future models will score exactly 85%
- Report the reversal and restrict the earlier inverse claim to its measured range
- Why does comparing unrelated small and large models complicate a claim that size caused a score change?
- Training data, objectives, and other choices may also differ
- Different model names make evaluation impossible
- The smaller model must have seen less data
- Parameter counts determine every training choice
- What can an easier distractor task cause in an inverse-scaling evaluation?
- The evaluation code must stop running
- A model may become better at the distracting pattern instead of the intended task
- All answer options become equally correct
- The true task is automatically solved
- Which information should be preserved so another team can check the evaluation?
- Only the largest model’s score
- Only a summary claiming bigger is worse
- Prompts, answer format, scoring code, model identifiers, and tested sizes
- Only the number of generated tokens
- If a model’s loss falls as scale increases, can that decline alone be called inverse scaling?
- Yes, every downward curve is inverse scaling
- Yes, lower loss means fewer correct predictions by definition
- Yes, model size determines the direction of every metric
- No; lower loss normally means improvement, so metric direction matters
Keep testing yourself
More quizzes picked for your level.