引導式測驗 · 中等 等級
語言模型測驗中的逆縮放
以任務分數、模型系列控制、記憶模式、幹擾因素、不確定性和可能的趨勢逆轉來解釋逆標度。
問題 1 的 8
三個規模越大、相似系列的模型在同一準確度任務上分別得分 80%、70% 和 60%。這說明了什麼?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 三個規模越大、相似系列的模型在同一準確度任務上分別得分 80%、70% 和 60%。這說明了什麼?
- An inverse trend on that task over the tested sizes
- That every capability declines with model size
- That all three models have identical training loss
- That a larger fourth model must score below 60%
- 為什麼平均下一個 token 損失更好卻會在指令跟隨任務上表現更差?
- The two measures always evaluate the same objective
- Predicting familiar text and following a changed instruction can favor different outputs
- Lower next-token loss always means worse language modeling
- Task accuracy is independent of the model’s output
- 提示提供了一個不尋常的結尾給熟悉的短語,但模型仍使用熟悉的結尾。這說明了哪種失敗模式?
- An increase in accelerator memory
- A reduction in vocabulary size
- A memorized continuation taking precedence over the supplied instruction
- A guaranteed improvement in task accuracy
- 第四個更大的模型在 80%、70% 和 60% 的序列之後得分 85%。評估報告應如何呈現?
- Discard the fourth result to preserve the original claim
- Declare that the earlier measurements never happened
- Predict all future models will score exactly 85%
- Report the reversal and restrict the earlier inverse claim to its measured range
- 為什麼比較無關的較小與較大模型會使聲稱尺寸導致分數變化變得複雜?
- Training data, objectives, and other choices may also differ
- Different model names make evaluation impossible
- The smaller model must have seen less data
- Parameter counts determine every training choice
- What can an easier distractor task cause in an inverse-scaling evaluation?
- The evaluation code must stop running
- A model may become better at the distracting pattern instead of the intended task
- All answer options become equally correct
- The true task is automatically solved
- Which information should be preserved so another team can check the evaluation?
- Only the largest model’s score
- Only a summary claiming bigger is worse
- Prompts, answer format, scoring code, model identifiers, and tested sizes
- Only the number of generated tokens
- If a model’s loss falls as scale increases, can that decline alone be called inverse scaling?
- Yes, every downward curve is inverse scaling
- Yes, lower loss means fewer correct predictions by definition
- Yes, model size determines the direction of every metric
- No; lower loss normally means improvement, so metric direction matters