Statistical Significance in LLM Evals

An evaluation score is an estimate based on a sample. If two prompts or models score differently, the gap may reflect a real performance difference, sample variation, or randomness in generation and judging. Statistical significance testing asks whether an observed difference would be surprising under a specified null hypothesis; it does not establish practical importance or guarantee a better product. When two systems are run on the same evaluation items, their results are paired. Paired bootstrap resampling or a paired test can preserve item-level correspondence and estimate uncertainty in the score difference. For classification-style outcomes on the same items, McNemar’s test is one possible method. Choice depends on the metric, data, and experimental design. The ACL tutorial by Dror and colleagues discusses this selection problem for NLP tasks. Model outputs can also vary across repeated runs, especially with sampling or changing backends. Use repeated runs when run-to-run variation is part of the target behavior, and distinguish that uncertainty from sampling uncertainty over examples. Report confidence intervals, the estimated effect size, sample size, and test procedure. If many prompts, models, metrics, or subgroups are compared, account for multiple comparisons or treat exploratory findings as provisional. Even a statistically significant change may be too small to matter, may hide subgroup regressions, or may be an artifact of a narrow test set. Define a practical threshold before testing, use held-out representative examples, and examine errors directly. Statistical evidence informs a release decision; it does not replace product judgment, safety review, or continuous monitoring.