แบบทดสอบที่เชื่อมโยงกับคำแนะนำ · ปานกลาง ระดับ
แบบทดสอบความไม่เท่าเทียมกันของ Tokenization ข้ามภาษา
ทดสอบว่าเหตุใดโทเค็นไนเซอร์จึงแบ่งบางภาษาออกเป็นโทเค็นมากขึ้น ผลกระทบต่อต้นทุนและกรอบเวลาบริบทอย่างไร และวิธีตรวจสอบและลดช่องว่าง
คำถาม 1 ของ 8
What is the main cause of large token-count gaps between languages?
ดูตัวอย่างทุกคำถาม8 คำถาม
คำตอบและคำอธิบายจะปรากฏขึ้นเมื่อคุณทำแบบทดสอบ
- What is the main cause of large token-count gaps between languages?
- Some languages are naturally ten times wordier than English
- Tokenizers were built from training text dominated by English and a few other languages
- Non-English text is compressed before tokenizing
- Models charge extra fees for translation
- Why do scripts like Devanagari or Thai often produce more tokens under byte-based tokenizers?
- Their characters usually take three bytes each in UTF-8 and may be rarely merged
- They are always written without spaces
- They are converted to images first
- Tokenizers reject them and replace them with English
- Which of these is NOT one of the three effects of higher token counts named in the guide?
- Higher cost per message
- Less text fitting in the context window
- Slower generation from more steps
- Automatic refusal to answer
- What does the term fertility mean in this context?
- How quickly a model learns a new language
- The average number of tokens produced per word
- The number of languages a model supports
- ขนาดของชุดข้อมูลการฝึกอบรม
- What change did OpenAI describe when it released GPT-4o in 2024?
- It stopped charging for non-English text
- It removed its tokenizer entirely
- It introduced a larger tokenizer that used fewer tokens for many non-English languages
- It limited GPT-4o to English only
- What is the best first step to plan costs for a multilingual app?
- Assume every language costs the same as English
- Tokenize representative translated documents with your model's exact tokenizer and compare counts
- Pick the cheapest model regardless of language
- Translate all user messages into English first without testing
- What tradeoff comes with making a tokenizer's vocabulary larger?
- It makes English text impossible to process
- It increases the size of the embedding and output layers
- โดยไม่จำเป็นต้องใช้ข้อมูลการฝึกอบรมอีกต่อไป
- It always lowers model quality
- According to the guide, why is 'some languages are just wordier' a misconception?
- All translations are exactly the same length
- Wordiness has no effect on anything
- Large token gaps come mainly from tokenizer design, not from the languages themselves
- Tokenizers only count spaces