Викторина по токенизации подслов

Ответы и пояснения появляются по мере прохождения теста.

1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
   • Every smaller piece necessarily has one unique dictionary meaning
   • A finite word-only vocabulary cannot allocate an entry to every possible word form
   • A subword model requires no numerical input representation
   • Whole-word vocabularies cannot contain any common word
2. Which subword tokenizer does the original BERT use?
   • A word-only alphabetical index
   • A mandatory one-byte-per-word lookup
   • WordPiece
   • An image-pixel segmentation network
3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
   • ##
   • A required trailing full stop
   • A leading plus sign on every whole word
   • Capitalization of every suffix
4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
   • It requires every word to have spaces on both sides
   • It translates every sentence into English before segmentation
   • It deletes all non-Latin characters before training
   • It can train from raw sentences without language-specific word pre-tokenization
5. Во что в конечном итоге сопоставляется каждый фрагмент подслова, прежде чем он достигнет модели?
   • Цвет
   • Отдельная нейронная сеть
   • Целочисленный идентификатор
   • Аудиоклип
6. If related word forms share an available vocabulary piece, what can that sharing support?
   • A guarantee that every token is a correct linguistic morpheme
   • Reuse of the representation associated with that piece
   • Deletion of every differing suffix from the original input
   • A requirement that all related words receive the same complete ID sequence
7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
   • It requires a human dictionary definition for every piece
   • It only merges the most frequent pair until the corpus is one token
   • It permanently removes all character-level coverage regardless of configuration
   • It starts with candidate pieces and prunes them according to an objective
8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
   • It is a library that supports both Unigram and BPE models
   • It is the name of BERT's ## continuation marker
   • It is a universal ID-to-word table shared by every model
   • It only implements whole-word splitting on English spaces