Using Logprobs for LLM Confidence

A log probability is the logarithm of a probability. In generation, a token logprob describes how likely the model assigned a particular next token given the preceding context and model state. Some APIs can return logprobs for generated tokens and, in some endpoints, alternatives among the most likely tokens. Availability and limits depend on provider, endpoint, and model. These values can help inspect ambiguity or compare candidate labels in a constrained classification task. But natural language answers consist of multiple tokens, and tokenization, answer length, wording, and prompt design affect sequence scores. A high probability for each token does not mean the answer is factually correct; models may confidently generate common but false continuations. Summing or averaging token logprobs also needs careful normalization and validation. Confidence calibration asks whether examples assigned a confidence level are correct at a corresponding frequency. To use logprobs as a confidence feature, define the target outcome, gather representative labeled examples, derive a score, and evaluate calibration and error detection on held-out data. Possible measures include reliability diagrams, Brier score, expected calibration error, and selective risk at a chosen abstention threshold. Research on language-model calibration finds that token probabilities and verbalized confidence can behave differently across model families and tasks. Treat logprobs as one signal, not an answer-level truth meter. Use human review or a fallback for high-impact decisions, and check whether the required API actually exposes logprobs for the selected model.