GUIDE Sosiete

Tokenization wuute ci làkk yi

Tokenization disparities mean the same message can take many more tokens in some languages than in English, because most tokenizers were built from text that is mostly English.

Déglul tegtal biiIA-nettali · ~3 min

Ci xët wii4 simili jàng
  1. Résumé
  2. Plongeur bu xóot
  3. Doxal ci àdduna dëgg
  4. The Future of Tokenization Disparities Across Languages
  5. Teg ko ci jëf
  6. Weyal di banneexu
  7. Laaj yi ñuy faral di laaj

Résumé

Since AI services charge by the token and limit how many tokens fit in a context window, speakers of those languages can pay more, fit less text and wait longer for the same work. That makes tokenizer design a fairness issue as well as a technical one.

Plongeur bu xóot

A tokenizer learns which chunks of text to treat as single tokens by studying a training corpus. Common strings get their own tokens; rare strings are broken into smaller pieces, sometimes down to individual bytes. When that corpus is dominated by English and a few other high-resource languages, those languages get efficient tokens for whole words and common word parts. Languages that appeared less often, or that use scripts where each character takes several bytes in UTF-8 encoding, such as many South Asian, Southeast Asian and African scripts, get split into many more pieces. The effect is measurable. Research published in 2023, including work by Petrov and colleagues on tokenizer unfairness, found that translations of the same text could need several times more tokens in some languages than in English, with the largest gaps exceeding ten times for certain languages and tokenizers. This matters in three concrete ways. Cost: API pricing is per token, so a longer token count is a direct price increase for the same content. Context: a fixed context window holds less actual text, so documents must be cut down and conversations lose earlier turns sooner. Speed and quality: more tokens means more generation steps, and heavy fragmentation can make it harder for the model to learn good representations of words, which may contribute to weaker performance in those languages. Providers have responded partly. When OpenAI released GPT-4o in 2024, it introduced a larger tokenizer and said it used fewer tokens for many non-English languages. Larger vocabularies and more balanced training data help, but they do not remove the gap entirely. A common misconception is that some languages are simply wordier. Translations do differ in length, but the large token gaps come mainly from how the tokenizer was built, not from the languages themselves.

Doxal ci àdduna dëgg

A health nonprofit translates the same patient leaflet into English, Hindi and Amharic and finds the non-English versions use several times more tokens, so its API bill for those languages is much higher.

A chatbot for Burmese speakers hits the context limit after far fewer conversation turns than the English version, so it forgets earlier messages sooner.

A developer checks a Korean support article with a tokenizer viewer and sees many words split into short fragments, while the English original mostly maps to one token per word.

A research team compares an older and a newer tokenizer from the same provider and finds the newer one uses noticeably fewer tokens for their Arabic and Tamil documents.

The Future of Tokenization Disparities Across Languages

Tokenizer vocabularies have been growing, and several providers now publicize improvements for non-English text, so the gap is likely to keep narrowing for widely spoken languages. Low-resource languages may continue to lag unless training data and tokenizer design deliberately include them. Research into byte-level and dynamically grouped models could reduce dependence on a fixed vocabulary, though those designs have their own compute costs. For now, organizations serving multilingual audiences should measure token counts in each language they support and treat per-language cost and context limits as part of accessibility planning.

Teg ko ci jëf

Lu tax mu am solo

  • Risk ak kaaraange. Gaañ-gaañu IA yu mag yi ak yu bës bu nekk yépp a ngi aju ci ki xam risk yi ak ki mëna def dara.
  • dogal yu gëna leer. Liggéeyukaay ak xam-xam bu ñépp bokk mooy wane ndax politiku kaaraange bu dëgër mën na am ci wàllu politik.
  • Dagg ci hype. Faram-fàcce yu leer dañuy wàññi li ñuy jàpp ci hype, PR lab, ak tiyaatar bu leerul.

Moytul

  • Jàppale risku nekk gi ni siyaas fiksioŋ fekk kàttan gi dafay yokk.
  • Jaxasoo kaaraange produit surface ak jubluwaay ci suufu autonomie bu kawe.
  • Bàyyi nit ñi xamul làkku Àngle ak ñi xamul làkku Angale, ñu am balluwaay yu baaxul.

Jéego yi ci topp

  1. Tàqale loraange yi ci produit bi, jëfandikoo bu baaxul, ak risku ñàkka mëna yor / ñàkka méngoo.
  2. Laajteel ban firnde mooy soppi sa xalaat ci kalendriye yi ak tar gi.
  3. Danga taamu balluwaay yu njëkk yi ak jàngat yu fëgër yi moo gën waxtaanu njaay mi.
  4. Xaarandil benn yoonu jëf: liggéey, politik, xaalis, wala xam-xam — du xam-xam kese.

Weyal di banneexu

Bulletin bu amul fayda

Jàngal xibaar AI bu bes

Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.

Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.

Nattal sa bopp

Take the Tokenization Disparities Across Languages quiz

Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.

Tambalil quiz

Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax

Laaj yi ñuy faral di laaj

What is Tokenization Disparities Across Languages?

Tokenization disparities mean the same message can take many more tokens in some languages than in English, because most tokenizers were built from text that is mostly English. Since AI services charge by the token and limit how many tokens fit in a context window, speakers of those languages can pay more, fit less text and wait longer for the same work. That makes tokenizer design a fairness issue as well as a technical one.

Lan mooy sabab bu mag bi am ci diggante lakk yi?

Tokenizers yi dañuy jox token yu baax ci buum yiñ gëna xam ci seen corpus tàggat yaram. Làkk yiñ joxul cëram dañuy xaajaloo nekk pàcc yu gëna ndaw.

Lan moo waral script yu melni Devanagari wala Thai dañuy faral di defar ay jeton yu bari ci suufu ay jeton yu lalu ci byte?

Bépp araf ci script yii ay byte UTF-8 la. Sudee tokenizer bi bariwul lumuy boole byte yooyu, benn araf mën na nekk ay token yu bari.

Ci yii, ban ci ñoom moo nekkul benn ci ñatti njeexte yi ci lim token yu gëna mag yiñ wax ci tegtal bi?

Gid bi dafay lim njëg yi, bayaali mbir yi, gaawaay bi wala kalite bi. Bañ ci saasi bokkul ak lim token.

Luy tekki ci kàddu njuram ci misaal bii?

Njur gi dafay natt token ci baat bu nekk. Fertilite bu gëna rëy mooy mbind mi dafay xaajaloo ay pàcc yu bari te njëg bi dafay gëna yokk.

Ban coppite la OpenAI wax bimu génnee GPT-4o ci 2024?

GPT-4o dafa ànd ak tokenizer bu gëna mag bu OpenAI wax ni moo gëna am njariñ ci làkk yu bari yu dul Àngle.