概述
It is a learned proxy, not a direct measure of human values.
深入探讨
In RLHF-style pipelines, a reward model is often trained on human comparisons of model outputs. A pairwise preference loss encourages the preferred response to score above the rejected response. At inference, best-of-N may sample responses and select the highest-scoring candidate. Optimizing a proxy can lead to reward hacking when the score is imperfect. InstructGPT collected human rankings of four to nine model responses for a prompt and converted those rankings into pairwise preferences for reward-model training. A common Bradley–Terry preference model converts differences between scalar response scores into preference probabilities, and the pairwise loss trains preferred responses to score higher. A common reward-model implementation uses a scalar sequence-scoring head, though architectures vary. Ensembles may reveal scorer disagreement but cannot remove shared model errors or guarantee against reward hacking.
现实世界的实施
Train from pairwise response rankings.
Use scores to rank candidate responses in best-of-N selection.
Test reward predictions on held-out comparisons.
Inspect for outputs that score highly but violate the intended behavior.
奖励模型的未来
Reward models remain useful components of some alignment pipelines, but require ongoing evaluation and safeguards against optimizing unintended behaviors.
付诸实践
为什么这很重要
- 速度与规模. 语言工作流程可以在不牺牲一致性的情况下更快地移动。
- 交通与覆盖范围. 它扩展了跨语言和沟通方式的访问。
- 更清晰的判决. 团队可以花更多时间进行判断，而自动化则可以处理重复。
留意
- 幻觉的事实可以悄悄地进入报告、支持流程或研究成果。
- 及时的敏感性可能会在类似的请求中产生不一致的结果。
- 如果访问控制薄弱，敏感文本数据可能会暴露。
后续步骤
- 在推出之前定义输出格式、语气和质量标准。
- 当准确性很重要时，请使用可信来源进行地面响应。
- 为高风险输出保留人工审查检查点。
- 跟踪故障模式并定期重新训练提示或工作流程。
不断探索
常见问题
什么是奖励模型？
A reward model maps a prompt and response to a scalar score intended to approximate preference or another objective. It is a learned proxy, not a direct measure of human values.
对于给定的提示响应对，奖励模型的主要输出是什么？
奖励模型将提示和响应映射到代表预测质量或人类偏好的一个标量数字。
What feedback format did the InstructGPT paper collect to train its reward model?
InstructGPT labelers ranked sets of four to nine model outputs for a prompt; the rankings were converted into pairwise comparisons for reward-model training. This is specific to that paper’s procedure.
标准奖励模型损失优化了什么？
In the Bradley–Terry model, scalar-score differences parameterize the probability that one response is preferred to another, and the pairwise loss trains scores accordingly. These scores are not automatically calibrated measures of quality.
什么是“奖励黑客”？
当模型发现不完美的奖励模型评价很高但人类却不会的捷径（例如过长或奉承）时，奖励黑客就会发生。
奖励模型在架构上是如何从语言模型派生出来的？
In a common reward-model setup, a language-model backbone is adapted with a scalar scoring head that produces one score for a prompt-response sequence. This describes that setup, not every reward-model architecture.