Perplexity と言語メトリクス

An autoregressive language model assigns probabilities to tokens conditioned on their preceding context. Perplexity transforms their average negative log-likelihood into an effective branching-factor interpretation. With natural logarithms, it is the exponential of that average loss. Under a uniform distribution over 10,000 possible tokens at every scored step, perplexity is 10,000; assigning probability one to each reference token yields one. These are mathematical illustrations, not performance claims about a deployed model. Lower perplexity means a lower mean negative log-probability for the reference text, rather than necessarily more correct top-choice predictions or more helpful generated answers. Comparisons also require an appropriate common evaluation. Tokenization changes the units over which the loss is averaged, and different corpora, preprocessing, or context access can change scores. A finite-context model cannot condition on an unlimited prefix. Splitting a long text into disjoint chunks resets available context at chunk boundaries. A sliding or strided window instead retains preceding context for more prediction steps, with computational and bookkeeping tradeoffs. The same scored target should not be counted repeatedly merely because windows overlap. Conventional causal perplexity should also be distinguished from masked-model scoring: a bidirectional masked language model such as BERT does not directly define the same left-to-right sequence factorization. Separate scoring procedures would need to be identified. Other language metrics evaluate different outputs. BLEU uses modified n-gram precision with a brevity penalty for translation-reference comparisons. ROUGE supplies reference-overlap measures used for summaries. Neither overlap nor perplexity is a complete factuality or usefulness assessment.