แบบทดสอบที่เชื่อมโยงกับคำแนะนำ · ปานกลาง ระดับ

แบบทดสอบความไม่เท่าเทียมกันของ Tokenization ข้ามภาษา

ทดสอบว่าเหตุใดโทเค็นไนเซอร์จึงแบ่งบางภาษาออกเป็นโทเค็นมากขึ้น ผลกระทบต่อต้นทุนและกรอบเวลาบริบทอย่างไร และวิธีตรวจสอบและลดช่องว่าง

เส้นทางแนะนำที่เกี่ยวข้องความเหลื่อมล้ำของ Tokenization ในภาษาต่างๆ
คำถาม 1 ของ 8

What is the main cause of large token-count gaps between languages?

ดูตัวอย่างทุกคำถาม8 คำถาม

คำตอบและคำอธิบายจะปรากฏขึ้นเมื่อคุณทำแบบทดสอบ

  1. What is the main cause of large token-count gaps between languages?
    • Some languages are naturally ten times wordier than English
    • Tokenizers were built from training text dominated by English and a few other languages
    • Non-English text is compressed before tokenizing
    • Models charge extra fees for translation
  2. Why do scripts like Devanagari or Thai often produce more tokens under byte-based tokenizers?
    • Their characters usually take three bytes each in UTF-8 and may be rarely merged
    • They are always written without spaces
    • They are converted to images first
    • Tokenizers reject them and replace them with English
  3. Which of these is NOT one of the three effects of higher token counts named in the guide?
    • Higher cost per message
    • Less text fitting in the context window
    • Slower generation from more steps
    • Automatic refusal to answer
  4. What does the term fertility mean in this context?
    • How quickly a model learns a new language
    • The average number of tokens produced per word
    • The number of languages a model supports
    • ขนาดของชุดข้อมูลการฝึกอบรม
  5. What change did OpenAI describe when it released GPT-4o in 2024?
    • It stopped charging for non-English text
    • It removed its tokenizer entirely
    • It introduced a larger tokenizer that used fewer tokens for many non-English languages
    • It limited GPT-4o to English only
  6. What is the best first step to plan costs for a multilingual app?
    • Assume every language costs the same as English
    • Tokenize representative translated documents with your model's exact tokenizer and compare counts
    • Pick the cheapest model regardless of language
    • Translate all user messages into English first without testing
  7. What tradeoff comes with making a tokenizer's vocabulary larger?
    • It makes English text impossible to process
    • It increases the size of the embedding and output layers
    • โดยไม่จำเป็นต้องใช้ข้อมูลการฝึกอบรมอีกต่อไป
    • It always lowers model quality
  8. According to the guide, why is 'some languages are just wordier' a misconception?
    • All translations are exactly the same length
    • Wordiness has no effect on anything
    • Large token gaps come mainly from tokenizer design, not from the languages themselves
    • Tokenizers only count spaces