トークン化

A tokenizer supplies the discrete units used to represent text for a model. Depending on its configuration, a unit can correspond to a whole word, a subword piece, a character, or a byte-derived fragment. A token is therefore not necessarily one word or one visible character. Subword methods aim to balance a finite vocabulary with the ability to represent less common words through smaller pieces. BPE, WordPiece, and Unigram make different training or segmentation choices. Their output also depends on normalization and preprocessing, so an illustrative word split should not be treated as the output of every tokenizer. Each vocabulary item maps to an integer ID, which a model can use to obtain a numerical representation. Those IDs are meaningful within that vocabulary mapping. The same integer in two different tokenizers need not identify the same piece, so numerical equality alone does not establish compatibility with a model. Counting an encoded sequence gives an exact token count for the specified tokenizer and settings. A word count or a character-based rule of thumb is only an estimate. Multi-character pieces explain why token count differs from character count; this does not make character-level reasoning impossible or explain every model error by itself. The same text can segment differently across vocabularies, and language-specific coverage can affect sequence length. For an interface that measures context or usage in tokens, its own counting convention matters. Batch preparation adds other operations: padding adds designated filler positions to shorter sequences, while truncation removes positions to meet a length setting. Attention masks can identify padding positions for the model.