Quiz dotyczący różnic w tokenizacji w różnych językach

Odpowiedzi i wyjaśnienia pojawiają się w miarę rozwiązywania quizu.

1. Jaka jest główna przyczyna dużych różnic w liczbie tokenów między językami?
   • Some languages are naturally ten times wordier than English
   • Tokenizers were built from training text dominated by English and a few other languages
   • Non-English text is compressed before tokenizing
   • Models charge extra fees for translation
2. Dlaczego skrypty takie jak Devanagari lub Thai często generują więcej tokenów w ramach tokenizatorów opartych na bajtach?
   • Their characters usually take three bytes each in UTF-8 and may be rarely merged
   • They are always written without spaces
   • They are converted to images first
   • Tokenizers reject them and replace them with English
3. Który z nich NIE jest jednym z trzech efektów wyższej liczby tokenów wymienionych w przewodniku?
   • Higher cost per message
   • Less text fitting in the context window
   • Slower generation from more steps
   • Automatic refusal to answer
4. Co w tym kontekście oznacza termin płodność?
   • How quickly a model learns a new language
   • The average number of tokens produced per word
   • The number of languages a model supports
   • Rozmiar zbioru danych szkoleniowych
5. Jaką zmianę opisał OpenAI po wydaniu GPT-4o w 2024 roku?
   • It stopped charging for non-English text
   • It removed its tokenizer entirely
   • It introduced a larger tokenizer that used fewer tokens for many non-English languages
   • It limited GPT-4o to English only
6. What is the best first step to plan costs for a multilingual app?
   • Assume every language costs the same as English
   • Tokenize representative translated documents with your model's exact tokenizer and compare counts
   • Pick the cheapest model regardless of language
   • Translate all user messages into English first without testing
7. What tradeoff comes with making a tokenizer's vocabulary larger?
   • It makes English text impossible to process
   • It increases the size of the embedding and output layers
   • Eliminuje to potrzebę danych szkoleniowych
   • It always lowers model quality
8. According to the guide, why is 'some languages are just wordier' a misconception?
   • All translations are exactly the same length
   • Wordiness has no effect on anything
   • Large token gaps come mainly from tokenizer design, not from the languages themselves
   • Tokenizers only count spaces