가이드 연계 퀴즈 · 쉬움 레벨

하위 단어 토큰화 퀴즈

어휘 트레이드오프, BERT WordPiece, 연속 마커, SentencePiece 공백, 숫자 ID, 공유 조각, Unigram 가지치기 및 라이브러리를 테스트합니다.

관련 가이드 경로서브워드 토큰화
질문 1 의 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

모든 질문 미리보기8 질문

퀴즈를 풀면 정답과 설명이 나타납니다.

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. 각 하위 단어 조각은 모델에 도달하기 전에 궁극적으로 무엇에 매핑됩니까?
    • 색상
    • 별도의 신경망
    • 정수 ID
    • 오디오 클립
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces