Tokenization Disparities Across Languages

A tokenizer learns which chunks of text to treat as single tokens by studying a training corpus. Common strings get their own tokens; rare strings are broken into smaller pieces, sometimes down to individual bytes. When that corpus is dominated by English and a few other high-resource languages, those languages get efficient tokens for whole words and common word parts. Languages that appeared less often, or that use scripts where each character takes several bytes in UTF-8 encoding, such as many South Asian, Southeast Asian and African scripts, get split into many more pieces. The effect is measurable. Research published in 2023, including work by Petrov and colleagues on tokenizer unfairness, found that translations of the same text could need several times more tokens in some languages than in English, with the largest gaps exceeding ten times for certain languages and tokenizers. This matters in three concrete ways. Cost: API pricing is per token, so a longer token count is a direct price increase for the same content. Context: a fixed context window holds less actual text, so documents must be cut down and conversations lose earlier turns sooner. Speed and quality: more tokens means more generation steps, and heavy fragmentation can make it harder for the model to learn good representations of words, which may contribute to weaker performance in those languages. Providers have responded partly. When OpenAI released GPT-4o in 2024, it introduced a larger tokenizer and said it used fewer tokens for many non-English languages. Larger vocabularies and more balanced training data help, but they do not remove the gap entirely. A common misconception is that some languages are simply wordier. Translations do differ in length, but the large token gaps come mainly from how the tokenizer was built, not from the languages themselves.