Temperature and Sampling

A language model produces a score, called a logit, for each candidate token. Softmax converts these scores into probabilities, and a sampling procedure selects the next token. For positive temperature T, dividing logits by T changes the relative probability distribution before sampling. Values below one sharpen differences; values above one flatten them. If two logits differ by six, then division by temperature two gives a difference of three. That calculation describes the scores, not the truth of either candidate. Greedy decoding selects a highest-scoring token instead of sampling. It should be treated as a decoding rule rather than implemented by literally dividing logits by zero. A very small positive temperature concentrates probability but still differs conceptually from disabling sampling. Candidate filters offer another control. Top-k keeps a specified number of highest-probability tokens. Top-p, also called nucleus sampling, sorts candidates by probability and keeps the smallest leading set whose cumulative probability reaches or exceeds the chosen threshold. For probabilities 0.55, 0.30, 0.10, and 0.05 with top-p 0.80, the first two candidates are sufficient because they sum to 0.85. Their total need not equal the threshold exactly. These choices can influence diversity and repeatability, but no filter proves that an output is factual, safe, or syntactically valid. Generation behavior also depends on implementation and the rest of the decoding configuration, so record the actual settings when comparing results.