Language Modeling

Language modeling concerns probabilities of text sequences. In an autoregressive formulation, each next-token distribution is conditioned on earlier tokens, and the probability of a specified prefix is the product of its conditional token probabilities. An illustrative two-token prefix has probability one-half for its first token and one-fifth for its second token given the first. Their joint prefix probability is one-tenth. This calculation concerns the stated prefix and does not add an unstated end-of-sequence event. N-gram models approximate context with a fixed short history and estimate continuation probabilities from counts. Without appropriate smoothing, unseen continuations can receive zero probability; smoothing can change that behavior, so novelty is not a guarantee that every n-gram system fails. Neural models use learned representations and can provide a different context-dependent distribution. Transformer decoder attention uses masking to prevent access to later target positions in the stated autoregressive training setup. The next-token training target is already present in the observed text, providing a self-supervised prediction task. During teacher-forced training, the prefix for a target comes from the observed training sequence. During autoregressive generation, selected output tokens are appended and used as context for later predictions. Those are different sources of context even when they use the same learned model. Cross-entropy or negative log-likelihood rewards assigning probability to the observed target tokens. Sampling settings then affect how a continuation is selected from model scores. Temperature rescales a distribution and nucleus sampling selects a probability-mass subset. Changes to selection can affect output variety without certifying factuality, safety, or useful reasoning.