SentencePiece Tokenization Quiz

Answers and explanations appear as you take the quiz.

1. What key assumption does SentencePiece avoid that traditional tokenizers rely on?
   • That words are separated by spaces
   • That text is encoded in UTF-8
   • That sentences end with punctuation
   • That the vocabulary is fixed in advance
2. Why does SentencePiece replace spaces with a visible meta symbol?
   • To compress the text more aggressively
   • To make tokenization fully reversible (lossless)
   • To remove all punctuation automatically
   • To translate between languages
3. Which two algorithms does SentencePiece primarily support?
   • TF-IDF and Word2Vec
   • LSTM and CNN
   • BPE and the Unigram language model
   • K-means and PCA
4. How does the Unigram model build its vocabulary?
   • By starting empty and merging frequent character pairs
   • By copying an English dictionary
   • By using a fixed list of 1,000 words
   • By starting large and pruning the least useful pieces via EM
5. Which model famously uses a SentencePiece tokenizer?
   • T5
   • GPT-2
   • ELMo
   • GloVe
6. What advantage does byte-level fallback give SentencePiece?
   • It makes the model train faster
   • It guarantees any character can be represented, avoiding out-of-vocabulary failures
   • It removes the need for attention
   • It doubles the vocabulary size automatically
7. What kind of input can SentencePiece use to train a subword model directly?
   • Only text already converted to English word tokens
   • Only a fixed dictionary of complete words
   • Raw sentences, without first splitting them into word sequences
   • Only character-frequency tables with no sentence data
8. Why is a language-independent tokenizer useful for multilingual text?
   • It translates every input into one language before tokenizing
   • It assigns the same meaning to every token in every language
   • It removes the need for a model vocabulary
   • The same tokenizer pipeline can process scripts without requiring language-specific word segmentation