Online vs Offline Model Evaluation

Offline evaluation uses a fixed dataset to compare candidate behavior. It is usually easier to reproduce, cheaper and lower risk than serving a candidate to users. Metrics may include accuracy, ranking quality, calibration, robustness, fairness slices and latency on a test environment. Offline testing supports screening and regression checks, but its conclusions depend on data relevance, labeling quality and how candidates were generated.

Historical logs reflect the policy and population that created them. In recommendation or search, users only interact with items they were shown. A new model may rank candidates differently, creating outcomes not represented in the old logs. Labels can also be selectively observed, delayed or influenced by prior decisions. An offline score therefore answers a conditional question about the available evaluation data, not necessarily the causal effect of deploying a new system.

Online evaluation measures behavior in a live environment, often through a randomized controlled experiment, canary or other controlled rollout. It can capture the full product response, including user adaptation, workflow effects and system load. Online tests introduce risks: users may experience a worse variant, metrics can be noisy, and experiment design must address sample ratio, interference, novelty and guardrails. A/B testing should follow power and stopping rules.

A sound process uses both. Offline checks reject broken or clearly inferior candidates before user exposure. A limited online test then measures causal impact under stated randomization and eligibility assumptions. Define primary and guardrail metrics, segment analyses, experiment duration and rollback criteria in advance. Monitor operational outcomes and delayed labels. Offline-online disagreement is informative: it may reveal distribution shift, logging bias, metric mismatch or an unintended product effect. Neither setting alone proves universal quality. Report the population, evaluation window, candidate version and uncertainty to clarify what each result supports.