引导式测验 · 简单 等级

子词标记化测验

测试词汇权衡、BERT WordPiece、延续标记、SentencePiece 空白、数字 ID、共享片段、Unigram 剪枝和库。

相关引导路径子词标记化
问题 1 的 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

预览每个问题8 问题

当您参加测验时，就会出现答案和解释。

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. 每个子词片段在到达模型之前最终映射到什么？
    • 一种颜色
    • 单独的神经网络
    • 整数 ID
    • 音频剪辑
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces