Κουίζ που συνδέεται με οδηγό · Εύκολο Επίπεδο

Κουίζ υπολέξεων Tokenization

Δοκιμάστε ανταλλαγές λεξιλογίου, BERT WordPiece, δείκτες συνέχειας, κενό διάστημα SentencePiece, αριθμητικά αναγνωριστικά, κοινόχρηστα κομμάτια, κλάδεμα Unigram και βιβλιοθήκες.

Σχετικές διαδρομές οδηγώνTokenization υπολέξεων
Ερώτηση 1 του 8

Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?

Προεπισκόπηση κάθε ερώτησης8 ερωτήσεις

Οι απαντήσεις και οι εξηγήσεις εμφανίζονται καθώς κάνετε το κουίζ.

  1. Which vocabulary limitation motivates representing rare word forms through smaller reusable pieces?
    • Every smaller piece necessarily has one unique dictionary meaning
    • A finite word-only vocabulary cannot allocate an entry to every possible word form
    • A subword model requires no numerical input representation
    • Whole-word vocabularies cannot contain any common word
  2. Which subword tokenizer does the original BERT use?
    • A word-only alphabetical index
    • A mandatory one-byte-per-word lookup
    • WordPiece
    • An image-pixel segmentation network
  3. Which prefix marks word-internal continuation pieces in BERT's published WordPiece convention?
    • ##
    • A required trailing full stop
    • A leading plus sign on every whole word
    • Capitalization of every suffix
  4. What SentencePiece input-handling property supports text without explicit whitespace word boundaries?
    • It requires every word to have spaces on both sides
    • It translates every sentence into English before segmentation
    • It deletes all non-Latin characters before training
    • It can train from raw sentences without language-specific word pre-tokenization
  5. Σε τι αντιστοιχεί τελικά κάθε θραύσμα υπολέξεων πριν φτάσει στο μοντέλο;
    • Ένα χρώμα
    • Ένα ξεχωριστό νευρωνικό δίκτυο
    • Ένα ακέραιο αναγνωριστικό
    • Ένα ηχητικό κλιπ
  6. If related word forms share an available vocabulary piece, what can that sharing support?
    • A guarantee that every token is a correct linguistic morpheme
    • Reuse of the representation associated with that piece
    • Deletion of every differing suffix from the original input
    • A requirement that all related words receive the same complete ID sequence
  7. How does Unigram vocabulary training differ from BPE's bottom-up pair merging?
    • It requires a human dictionary definition for every piece
    • It only merges the most frequent pair until the corpus is one token
    • It permanently removes all character-level coverage regardless of configuration
    • It starts with candidate pieces and prunes them according to an objective
  8. Why should SentencePiece not be treated as another name for the Unigram algorithm?
    • It is a library that supports both Unigram and BPE models
    • It is the name of BERT's ## continuation marker
    • It is a universal ID-to-word table shared by every model
    • It only implements whole-word splitting on English spaces