Pairwise and Elo Evaluation

Absolute scores can be difficult to compare when raters or model judges use a numeric scale differently; some setups also show score compression. Pairwise evaluation reduces reliance on a shared numeric scale by asking which of two outputs for the same input is better, but it still has order, sampling, and preference biases. This is the same underlying approach used in platforms like Chatbot Arena, where users compare two anonymous model responses and vote for a preference, and those millions of individual votes feed into a rating system. Elo, originally developed for chess ratings, is one familiar aggregation method: ratings change based on the result relative to the expected win probability. Bradley-Terry is a related statistical model that estimates relative strengths from pairwise outcomes. Chatbot Arena initially used online Elo and later adopted Bradley-Terry for its rankings and uncertainty estimates; other projects choose a method to fit their assumptions. A common misconception is that Elo ratings from a small number of comparisons are precise; in practice, ratings have real uncertainty (often reported with confidence intervals) until enough comparisons accumulate, and comparisons should ideally be randomized in order and anonymized to avoid position or branding bias. Pairings also matter: models that rarely face the same opponents are harder to compare reliably, and new or changing model versions weaken the assumption that ratings describe static competitors.