Guide-linked quiz · Medium Level

Inverse Scaling in Language Models Quiz

Interpret inverse scaling with task scores, model-series controls, memorized patterns, distractors, uncertainty, and possible trend reversals.

Question 1 of 8

Three increasingly large models in a comparable series score 80%, 70%, and 60% on the same accuracy task. What does this show?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. Three increasingly large models in a comparable series score 80%, 70%, and 60% on the same accuracy task. What does this show?
    • An inverse trend on that task over the tested sizes
    • That every capability declines with model size
    • That all three models have identical training loss
    • That a larger fourth model must score below 60%
  2. Why can better average next-token loss coexist with worse performance on an instruction-following task?
    • The two measures always evaluate the same objective
    • Predicting familiar text and following a changed instruction can favor different outputs
    • Lower next-token loss always means worse language modeling
    • Task accuracy is independent of the model’s output
  3. A prompt supplies an unusual ending to a familiar phrase, but the model uses the familiar ending. Which proposed failure pattern does this illustrate?
    • An increase in accelerator memory
    • A reduction in vocabulary size
    • A memorized continuation taking precedence over the supplied instruction
    • A guaranteed improvement in task accuracy
  4. A fourth, larger model scores 85% after the sequence 80%, 70%, and 60%. How should the evaluation report this?
    • Discard the fourth result to preserve the original claim
    • Declare that the earlier measurements never happened
    • Predict all future models will score exactly 85%
    • Report the reversal and restrict the earlier inverse claim to its measured range
  5. Why does comparing unrelated small and large models complicate a claim that size caused a score change?
    • Training data, objectives, and other choices may also differ
    • Different model names make evaluation impossible
    • The smaller model must have seen less data
    • Parameter counts determine every training choice
  6. What can an easier distractor task cause in an inverse-scaling evaluation?
    • The evaluation code must stop running
    • A model may become better at the distracting pattern instead of the intended task
    • All answer options become equally correct
    • The true task is automatically solved
  7. Which information should be preserved so another team can check the evaluation?
    • Only the largest model’s score
    • Only a summary claiming bigger is worse
    • Prompts, answer format, scoring code, model identifiers, and tested sizes
    • Only the number of generated tokens
  8. If a model’s loss falls as scale increases, can that decline alone be called inverse scaling?
    • Yes, every downward curve is inverse scaling
    • Yes, lower loss means fewer correct predictions by definition
    • Yes, model size determines the direction of every metric
    • No; lower loss normally means improvement, so metric direction matters

Keep testing yourself

More quizzes picked for your level.

View all quizzes