AI Glossary Term

What does “Reward Model” mean?

Definition

A model that scores outputs based on preference signals, often used in RLHF pipelines.

Browse the full AI Glossary Explore all free AI guides

How Reward Model works

Example

A reward model trained on thousands of human rankings can give two draft replies different scores. During tuning, the chatbot is nudged toward replies that score higher.

Why it matters

It stands in for human judgment at scale. If the reward model is flawed, the tuned model can learn to exploit its flaws.

Common questions

How is it trained?

On examples where people chose the better of two or more answers.

Reward model vs human preference model?

They are closely related. A human preference model is one trained to predict what people prefer.

What is the risk?

Over-optimizing against it can produce answers that score highly but are not actually good.

Reinforcement Learning from Human Feedback (RLHF)A training method that uses human preference signals to shape model behavior.Reinforcement LearningTraining by reward signals where an agent learns actions that maximize long-term return.Ground TruthTrusted reference labels used to train or evaluate model outputs.DPO (Direct Preference Optimization)A training method that fine-tunes models directly on preference pairs without needing a separate reward model.Mid-trainingAn intermediate training phase between pretraining and post-training, often used for capability or domain adjustments.AI AgentA software system that can observe, reason, and take actions to achieve a goal, often using tools and memory.

Learn more in our free guides

AI Models Explained Model Collapse Model Lifecycle Model Context Protocol

Test yourself

AI Models Explained Quiz Model Collapse Quiz Model Lifecycle Quiz