Maswali ya Tokenization

Majibu na maelezo huonekana unapojibu maswali.

1. How should a language-model token be distinguished from a visible word or character?
   • It must always equal one complete English word
   • It is a tokenizer-defined unit that may span a word, subword, character, or byte-derived piece
   • It must always equal one visible Unicode character
   • It is necessarily an API access password
2. Which family includes BPE, WordPiece, and Unigram tokenization?
   • Only whole-sentence vocabularies
   • Only fixed image-pixel encodings
   • Subword tokenization methods
   • Only alphabetical word sorting
3. Why can a token count differ from the number of visible characters in the same text?
   • A token can contain several characters or byte-derived fragments
   • Every tokenizer deletes vowels before encoding
   • Every character must map to exactly ten IDs
   • Tokenization always replaces words with images
4. How should an exact input token count be obtained for a specified language model and encoding setup?
   • Multiply every word count by one fixed universal number
   • Count only sentence-ending periods
   • Assume every English word is exactly one token
   • Encode with its matching tokenizer and count the resulting IDs
5. For an interface measuring context capacity and usage in tokens, why does its tokenizer-specific count matter?
   • It guarantees all interfaces use identical pricing
   • It proves every input has the same word count
   • It determines how the input is counted under those token-based limits or usage rules
   • It changes only the displayed font
6. Why can comparable text segment into different numbers of pieces under different tokenizer vocabularies?
   • Every language necessarily has the same learned vocabulary
   • The available pieces and preprocessing can differ
   • Each tokenizer must use one ID for every sentence
   • Changing languages always leaves every segmentation unchanged
7. Why does an identical integer ID in two tokenizer vocabularies not establish model-input compatibility?
   • All vocabularies share a globally fixed ID-to-piece table
   • IDs themselves encode a universal dictionary definition
   • A vocabulary contains only token counts and no mappings
   • The two mappings can assign that number to different pieces
8. How does padding differ from truncation when preparing token sequences?
   • Padding adds filler positions; truncation removes positions under a length rule
   • Padding removes the longest sequence; truncation adds filler
   • Both operations necessarily select the most informative words
   • Both change only the numerical ID vocabulary without changing sequence length