引导式测验 · 中等 等级

拒绝采样微调测验

涵盖候选生成和过滤、基于验证者的奖励、监督再培训以及优化有缺陷的评分器的风险。

相关引导路径拒绝采样微调
问题 1 的 8

哪个循环最能描述拒绝采样微调如何改进模型？

预览每个问题8 问题

当您参加测验时，就会出现答案和解释。

  1. 哪个循环最能描述拒绝采样微调如何改进模型？
    • 生成许多候选答案，保留得分最高的答案，并对它们进行重新训练
    • 删除性能最差的模型层
    • 拒绝所有训练数据并重新开始
    • 采样一次并始终接受输出
  2. 在数学或代码等可验证领域，什么可以作为奖励？
    • 仅学习奖励模型
    • 答案是否正确或测试是否通过
    • 响应的长度
    • 生成的代币数量
  3. Which model family does the cited paper describe using rejection sampling in its alignment pipeline?
    • GPT-4o
    • BERT Base
    • Meta’s Llama 2-Chat
    • Whisper Large
  4. Why might a team choose rejection-sampling fine-tuning over a policy-gradient approach for some tasks?
    • It never needs a candidate generator or scoring rule
    • It guarantees no reward hacking
    • It always requires less compute than every RL method
    • It can use supervised learning on selected generations instead of updating through policy-gradient RL
  5. 对最大奖励样本进行有效训练有什么作用？
    • 将最好的 N 质量提取到模型的单样本输出中
    • 增加词汇量
    • 删除模型生成文本的能力
    • 将参数数量加倍
  6. What should teams monitor when repeatedly fine-tuning on filtered model-generated outputs?
    • Only the number of model parameters
    • Output diversity alongside held-out correctness
    • Whether every answer uses the same wording
    • Only the training loss on selected samples
  7. What is a limitation of rejection-sampling fine-tuning’s candidate selection?
    • It can only be used with image labels
    • It always changes the tokenizer
    • It cannot train on a high-quality answer that was never generated or retained
    • It makes the scoring function unnecessary
  8. What risk arises if the scoring model used to filter candidates is flawed?
    • The policy becomes unable to generate any text
    • The tokenizer automatically changes vocabulary
    • The reference labels become unnecessary
    • The policy may learn to exploit the scorer rather than improve the intended quality