AI Glossary Term
What does “Cross-Entropy Loss” mean?
A common objective function used to train classification models by penalizing incorrect probabilities.
How Cross-Entropy Loss works
Example
A classifier sees a photo of a dog. If it assigns 0.9 probability to 'dog', the loss is small; if it assigns only 0.1 to 'dog', the loss is large. Training adjusts the weights to push the probability on the correct class up.
Why it matters
Cross-entropy is the standard training signal for classifiers and language models. It punishes confident wrong answers more than uncertain ones, which encourages sensible probabilities.
Common questions
Why not just count correct answers?
Counting is not smooth, so it gives training no direction. Cross-entropy changes gradually with the probabilities, which makes optimization possible.
What does a loss of zero mean?
That the model gave probability 1 to the correct answer on every example, which on real data usually signals overfitting rather than perfection.
Is cross-entropy used for language models?
Yes. Language models are trained to minimize cross-entropy on predicting the next token, and perplexity is derived from it.