AI Glossary Term
What does “Tokenization” mean?
Definition
The process of splitting text into tokens for model input.
How Tokenization works
Example
Before a sentence reaches a language model, a tokenizer splits it into pieces and maps each to a number. The model only ever sees the numbers, and the output numbers are converted back to text.
Why it matters
Tokenization affects cost, speed, and sometimes ability, such as with spelling, arithmetic, or non-English text.
Common questions
Do all models use the same tokenizer?
No. Each model family has its own, so token counts differ.
How is the vocabulary decided?
Methods such as BPE learn common character sequences from a large text sample.
Why do models struggle with letters in a word?
They see tokens rather than individual letters, which makes tasks like counting letters harder.