Brier Score

A classification model may output both a label and a probability. A delivery system might call two parcels late while assigning probabilities of 0.6 and 0.9. Accuracy treats those predictions alike after a threshold converts them into labels. The Brier score preserves the difference in confidence. For a binary event, encode occurrence as 1 and nonoccurrence as 0. Subtract the outcome from the predicted probability, square the difference, and average across cases. In the commonly used binary convention, the result ranges from 0 to 1, with smaller values indicating less probability error. Always identify the event being predicted: a probability of arriving late cannot be compared with a label that means arriving on time. Consider two hypothetical parcels with late-arrival probabilities of 0.8 and 0.3. The first is late and the second is on time. Their losses are 0.04 and 0.09, producing an average of 0.065. These are arithmetic examples, not results from a deployed product or a research study. The score needs a meaningful comparison. A low score can be easy to achieve when the event almost never happens. Compare the model with a simple baseline on the same evaluation cases, and report the event rate. Do not claim that a particular score is universally good across unrelated datasets. Also inspect calibration: among cases assigned similar probabilities, how often does the event occur? A lower Brier score alone does not prove better calibration because the score also rewards separating cases with different risks. Scikit-learn documents both probability scoring and calibration tools. Together with an appropriate evaluation split, they help distinguish useful confidence from convincing-looking numbers.