Tokenization ati Baiti Tọki Ìbòmọlẹ adanwo

Awọn idahun ati awọn alaye han bi o ṣe n mu ibeere naa.

1. Kí ni àmi ń mú jáde fún àwòkọ́ṣe èdè kan láti kà?
   • Awọn ami-ami, eyiti o jẹ ID odidi ti a ya aworan si awọn ege ọrọ
   • Aise iwe igbi
   • Piksẹli akoj
   • Compiled koodu ẹrọ
2. Bawo ni fifi koodu Baiti Pair ṣe kọ awọn ọrọ rẹ?
   • Nipa fifi aami-ami kan fun gbogbo ọrọ ni iwe-itumọ
   • Nipa didapọ leralera awọn aami meji to sunmọ julọ loorekoore
   • Nipa ṣiṣe akojọpọ awọn ohun kikọ laileto
   • Nipa titumọ ọrọ si ede miiran ni akọkọ
3. Iṣoro wo ni awọn ọna-ọrọ bii BPE yanju ni akawe si isọdi ipele-ọrọ?
   • Wọn ṣe gbogbo gbolohun gangan aami kan
   • Wọn ṣe imukuro iwulo fun eyikeyi fokabulari
   • Wọ́n ń ka àwọn ọ̀rọ̀ tó ṣọ̀wọ́n tàbí àwọn ọ̀rọ̀ tí a kò lè rí nípa pípín wọ́n sọ́tọ̀ sí àwọn àjákù tí a tún lè lò
   • Wọn nikan ṣiṣẹ fun English
4. Kini anfani ti ipele baiti BPE, bi a ti lo ninu awọn awoṣe GPT?
   • O nlo awọn ami-ami diẹ sii ju eyikeyi ọna miiran fun gbogbo awọn ede
   • O yọ iwulo lati ṣe ikẹkọ awoṣe
   • O ṣe koodu awọn lẹta Gẹẹsi nikan
   • O le ṣe aṣoju ọrọ Unicode eyikeyi, pẹlu emoji ati ede eyikeyi, laisi awọn ikuna ti ita-fokabulari
5. Kini idi ti kika ami awoṣe awoṣe nigbagbogbo yatọ si nọmba awọn ọrọ ninu gbolohun ọrọ kan?
   • Awọn aaye, titobi nla, ati awọn ọrọ to ṣọwọn yipada bi ọrọ ṣe pinya, ati pe ọrọ kan le di awọn ami-ami pupọ
   • Awọn ami-ami ni a ka ni ede laileto
   • Ọrọ kọọkan jẹ awọn ami ami meji deede
   • Awọn iṣiro tokini nikan da lori awọn aami ifamisi
6. Lati inu aaye wo ni a ti ya Algoridimu koodu Baiti bata ni akọkọ bi?
   • Ṣiṣe aworan
   • Data funmorawon
   • Atọka aaye data
   • Cryptography
7. After BPE training creates its vocabulary, what does an encoder use to tokenize new text?
   • A new merge ranking learned separately for each sentence
   • A dictionary of whole words only
   • The learned vocabulary and ordered merge rules applied to the input symbols
   • A translation table that changes the text language
8. Why can byte-level BPE encode a character it never saw as a whole token during training?
   • The tokenizer invents new model parameters for each character
   • It drops every unseen character
   • The model automatically translates it to English first
   • The UTF-8 input can be represented as a sequence of bytes from a finite base alphabet