Maswali yanayounganishwa na mwongozo · Rahisi Kiwango

Maswali ya Tokeni ya Neno Ndogo

Jaribu mabadiliko ya msamiati, BERT WordPiece, vialamisho vya kuendelea, nafasi nyeupe ya SentencePiece, vitambulisho vya nambari, vipande vilivyoshirikiwa, kupogoa kwa Unigram na maktaba.

Njia za mwongozo zinazohusianaUwekaji Tokeni wa Neno Ndogo
Swali 1 ya 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

Hakiki kila swali8 maswali

Majibu na maelezo huonekana unapojibu maswali.

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. Je, kila kipande cha neno ndogo hatimaye huweka ramani ya nini kabla ya kufikia kielelezo?
    • Rangi
    • Mtandao tofauti wa neva
    • Kitambulisho kamili
    • Kipande cha sauti
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces