Quiz lié à un guide · Facile Niveau

Quiz sur la tokenisation des sous-mots

Testez les compromis de vocabulaire, BERT WordPiece, les marqueurs de continuation, les espaces SentencePièce, les identifiants numériques, les pièces partagées, l'élagage Unigram et les bibliothèques.

Chemins de guidage associésTokenisation de sous-mots
Question 1 de 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

Prévisualisez chaque question8 des questions

Les réponses et les explications apparaissent au fur et à mesure que vous répondez au quiz.

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. À quoi correspond finalement chaque fragment de sous-mot avant d’atteindre le modèle ?
    • Une couleur
    • Un réseau neuronal séparé
    • Un identifiant entier
    • Un extrait audio
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces