Cuestionario vinculado a una guía · fácil Nivel

Prueba de tokenización de subpalabras

Pruebe compensaciones de vocabulario, BERT WordPiece, marcadores de continuación, espacios en blanco de SentencePiece, identificaciones numéricas, piezas compartidas, poda de Unigram y bibliotecas.

Rutas de guía relacionadasTokenización de subpalabras
Pregunta 1 de 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

Vista previa de cada pregunta8 preguntas

Las respuestas y explicaciones aparecen a medida que realiza el cuestionario.

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. ¿A qué se asigna en última instancia cada fragmento de subpalabra antes de llegar al modelo?
    • un color
    • Una red neuronal separada
    • Una identificación entera
    • un clip de audio
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces