Guide-yakabatana mibvunzo · Pakati Nhanho

Tokenization uye Byte Pair Encoding Quiz

Tests token IDs, frequency-based BPE merges, byte-level coverage, merge order, and why token counts differ from word counts.

Nzira dzekutungamira dzinoenderanaTokenization uye Byte Pair Encoding
Mubvunzo 1 ye 8

Chii chinonzi tokenization chinoburitsa kune modhi yemutauro kuverenga?

Tarisa mubvunzo wese8 mibvunzo

Mhinduro uye tsananguro zvinoonekwa paunenge uchitora bvunzo.

  1. Chii chinonzi tokenization chinoburitsa kune modhi yemutauro kuverenga?
    • Tokeni, ari maID ID akamepurwa kuita zvidimbu zvemavara
    • Raw audio waveforms
    • Pixel grids
    • Yakagadzirwa muchina kodhi
  2. Ko Byte Pair Encoding inovaka sei mazwi ayo?
    • Nekupa chiratidzo chimwe pazwi rose muduramazwi
    • Nekudzokorodza kubatanidza iyo inowanzoitika padyo peya yezviratidzo
    • Nekungoisa mumapoka mavara
    • Nekutanga kushandura zvinyorwa kune mumwe mutauro
  3. Nderipi dambudziko rinogadziriswa ne subword nzira seBPE zvichienzaniswa neshoko-level tokenization?
    • Vanoita mutsara wega wega chiratidzo chimwe chete
    • Vanobvisa kudikanwa kweshoko ripi neripi
    • Vanobata mazwi asingawanzo kana asingaoneki nekuapatsanura kuita zvidimbu zvinogona kushandiswa zvakare
    • Vanoshanda kuChirungu chete
  4. Chii chakanakira byte-level BPE, sekushandiswa mumhando dzeGPT?
    • Inoshandisa zviratidzo zvishoma kupfuura chero imwe nzira yemitauro yose
    • Inobvisa kudiwa kwekudzidzisa muenzaniso
    • Inoisa mabhii echirungu chete
    • Inogona kumiririra chero mavara eUnicode, kusanganisira emoji uye chero mutauro, pasina kukundikana kwemazwi
  5. Sei nhamba yechiratidzo chemuenzaniso ichiwanzosiyana nehuwandu hwemashoko mumutsara?
    • Nzvimbo, capitalization, uye mazwi asingawanzo shandura kupatsanurwa kwemavara, uye izwi rimwe rinogona kuita tokeni akati wandei
    • Zviratidzo zvinoverengwa mumutauro usina kurongeka
    • Izwi rega rega rinogara rine ma tokeni maviri chaiwo
    • Zviverengero zvezviratidzo zvinoenderana chete nezviratidzo
  6. Kubva kumunda upi wakakweretwa iyo Byte Pair Encoding algorithm?
    • Kuburitsa mufananidzo
    • Data compression
    • Database indexing
    • Cryptography
  7. After BPE training creates its vocabulary, what does an encoder use to tokenize new text?
    • A new merge ranking learned separately for each sentence
    • A dictionary of whole words only
    • The learned vocabulary and ordered merge rules applied to the input symbols
    • A translation table that changes the text language
  8. Why can byte-level BPE encode a character it never saw as a whole token during training?
    • The tokenizer invents new model parameters for each character
    • It drops every unseen character
    • The model automatically translates it to English first
    • The UTF-8 input can be represented as a sequence of bytes from a finite base alphabet