Ci bi mu melGis bi ci topp
Jëfandikoo xel mu amul dëgg-dëgg ci diggante genn aju xel
Askan wi
GUIDE Sosiete
Tokenization disparities mean the same message can take many more tokens in some languages than in English, because most tokenizers were built from text that is mostly English.
Déglul tegtal biiIA-nettali · ~3 min
Since AI services charge by the token and limit how many tokens fit in a context window, speakers of those languages can pay more, fit less text and wait longer for the same work. That makes tokenizer design a fairness issue as well as a technical one.
A tokenizer learns which chunks of text to treat as single tokens by studying a training corpus. Common strings get their own tokens; rare strings are broken into smaller pieces, sometimes down to individual bytes. When that corpus is dominated by English and a few other high-resource languages, those languages get efficient tokens for whole words and common word parts. Languages that appeared less often, or that use scripts where each character takes several bytes in UTF-8 encoding, such as many South Asian, Southeast Asian and African scripts, get split into many more pieces. The effect is measurable. Research published in 2023, including work by Petrov and colleagues on tokenizer unfairness, found that translations of the same text could need several times more tokens in some languages than in English, with the largest gaps exceeding ten times for certain languages and tokenizers. This matters in three concrete ways. Cost: API pricing is per token, so a longer token count is a direct price increase for the same content. Context: a fixed context window holds less actual text, so documents must be cut down and conversations lose earlier turns sooner. Speed and quality: more tokens means more generation steps, and heavy fragmentation can make it harder for the model to learn good representations of words, which may contribute to weaker performance in those languages. Providers have responded partly. When OpenAI released GPT-4o in 2024, it introduced a larger tokenizer and said it used fewer tokens for many non-English languages. Larger vocabularies and more balanced training data help, but they do not remove the gap entirely. A common misconception is that some languages are simply wordier. Translations do differ in length, but the large token gaps come mainly from how the tokenizer was built, not from the languages themselves.
A health nonprofit translates the same patient leaflet into English, Hindi and Amharic and finds the non-English versions use several times more tokens, so its API bill for those languages is much higher.
A chatbot for Burmese speakers hits the context limit after far fewer conversation turns than the English version, so it forgets earlier messages sooner.
A developer checks a Korean support article with a tokenizer viewer and sees many words split into short fragments, while the English original mostly maps to one token per word.
A research team compares an older and a newer tokenizer from the same provider and finds the newer one uses noticeably fewer tokens for their Arabic and Tamil documents.
Tokenizer vocabularies have been growing, and several providers now publicize improvements for non-English text, so the gap is likely to keep narrowing for widely spoken languages. Low-resource languages may continue to lag unless training data and tokenizer design deliberately include them. Research into byte-level and dynamically grouped models could reduce dependence on a fixed vocabulary, though those designs have their own compute costs. For now, organizations serving multilingual audiences should measure token counts in each language they support and treat per-language cost and context limits as part of accessibility planning.
Bulletin bu amul fayda
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Tokenization disparities mean the same message can take many more tokens in some languages than in English, because most tokenizers were built from text that is mostly English. Since AI services charge by the token and limit how many tokens fit in a context window, speakers of those languages can pay more, fit less text and wait longer for the same work. That makes tokenizer design a fairness issue as well as a technical one.
Tokenizers yi dañuy jox token yu baax ci buum yiñ gëna xam ci seen corpus tàggat yaram. Làkk yiñ joxul cëram dañuy xaajaloo nekk pàcc yu gëna ndaw.
Bépp araf ci script yii ay byte UTF-8 la. Sudee tokenizer bi bariwul lumuy boole byte yooyu, benn araf mën na nekk ay token yu bari.
Gid bi dafay lim njëg yi, bayaali mbir yi, gaawaay bi wala kalite bi. Bañ ci saasi bokkul ak lim token.
Njur gi dafay natt token ci baat bu nekk. Fertilite bu gëna rëy mooy mbind mi dafay xaajaloo ay pàcc yu bari te njëg bi dafay gëna yokk.
GPT-4o dafa ànd ak tokenizer bu gëna mag bu OpenAI wax ni moo gëna am njariñ ci làkk yu bari yu dul Àngle.
Weyal di jàng
Tann nañu yeneen njiit ngir topic bii
Ci bi mu melGis bi ci topp
Jëfandikoo xel mu amul dëgg-dëgg ci diggante genn aju xel
Askan wi