Tokenization and Byte Pair Encoding Quiz

Answers and explanations appear as you take the quiz.

1. What does tokenization produce for a language model to read?
   • Tokens, which are integer IDs mapped to pieces of text
   • Raw audio waveforms
   • Pixel grids
   • Compiled machine code
2. How does Byte Pair Encoding build its vocabulary?
   • By assigning one token per whole word in a dictionary
   • By repeatedly merging the most frequent adjacent pair of symbols
   • By randomly grouping characters
   • By translating text into another language first
3. What problem do subword methods like BPE solve compared to word-level tokenization?
   • They make every sentence exactly one token
   • They eliminate the need for any vocabulary
   • They handle rare or unseen words by splitting them into reusable fragments
   • They only work for English
4. What is the advantage of byte-level BPE, as used in GPT models?
   • It uses fewer tokens than any other method for all languages
   • It removes the need to train the model
   • It only encodes English letters
   • It can represent any Unicode text, including emoji and any language, with no out-of-vocabulary failures
5. Why does a model's token count often differ from the number of words in a sentence?
   • Spaces, capitalization, and rare words change how text splits, and one word can become several tokens
   • Tokens are counted in a random language
   • Each word is always exactly two tokens
   • Token counts only depend on punctuation
6. From which field was the Byte Pair Encoding algorithm originally borrowed?
   • Image rendering
   • Data compression
   • Database indexing
   • Cryptography
7. After BPE training creates its vocabulary, what does an encoder use to tokenize new text?
   • A new merge ranking learned separately for each sentence
   • A dictionary of whole words only
   • The learned vocabulary and ordered merge rules applied to the input symbols
   • A translation table that changes the text language
8. Why can byte-level BPE encode a character it never saw as a whole token during training?
   • The tokenizer invents new model parameters for each character
   • It drops every unseen character
   • The model automatically translates it to English first
   • The UTF-8 input can be represented as a sequence of bytes from a finite base alphabet