AI Glossary Term

What does “Confidence Interval” mean?

Definition

A statistical range that likely contains the true value of a measured model metric.

Browse the full AI Glossary Explore all free AI guides

How Confidence Interval works

Example

A model scores 87 percent accuracy on a test set of 500 examples. A 95 percent confidence interval of roughly 84 to 90 percent shows that, with a test set this size, the true accuracy could plausibly be a few points either way.

Why it matters

A single score can overstate precision. Intervals show how much a result might change with different test data, which helps when comparing models.

Common questions

What does 95 percent mean?

If you repeated the measurement many times, about 95 percent of such intervals would contain the true value.

How do I make the interval narrower?

Use a larger test set.

If two intervals overlap, are the models the same?

Overlap suggests the difference may not be reliable, though formal tests on the paired results are more precise.

PerplexityA language-model metric measuring how surprised the model is by true next tokens.CalibrationHow well a model's confidence scores match actual correctness probabilities.ComputeThe processing resources required to train and run models, often measured in FLOPS or GPU hours.False NegativeAn incorrect prediction where a model misses a true positive case.HyperparameterA configuration value set before training, such as learning rate, batch size, or depth.Top-k SamplingA decoding strategy that samples only from the k most likely next tokens.

Learn more in our free guides

Using Logprobs for LLM Confidence Confidence Intervals ASR Confidence Scores and Calibration What AI Confidence Scores Actually Mean

Test yourself

Using Logprobs for LLM Confidence Quiz Confidence Intervals Quiz ASR Confidence Scores and Calibration Quiz