Tokenisation WordPièce

Tokenization is the first step in any language model: turning raw text into a sequence of units, called tokens, that the model has embeddings for. Splitting on spaces gives whole words, but no fixed word list can cover every name, typo, technical term and inflected form, and every unknown word collapses into a single unknown token. Splitting into characters covers everything but makes sequences long and forces the model to learn spelling from scratch. Subword tokenization sits between the two: common words stay whole, and rare words are broken into smaller pieces that are in the vocabulary. WordPiece is one of the main subword methods, alongside byte-pair encoding (BPE) and the unigram language model used in SentencePiece.

WordPiece comes from Google. Mike Schuster and Kaisuke Nakajima described it in a 2012 ICASSP paper on Japanese and Korean voice search, where text has no spaces between words and a large character set, so a speech recognizer needed a data-driven way to choose units. Google's 2016 neural machine translation paper by Yonghui Wu and colleagues adopted the same model and explained its goal: given a training corpus and a target number of tokens D, pick D wordpieces so the corpus can be segmented into as few pieces as possible, which amounts to maximizing the likelihood of the data under a simple language model. Its example sentence segments Jet makers feud over seat width as _J et _makers _fe ud _over _seat _width, where the underscore marks the start of a word and common words stay whole.

Training a WordPiece vocabulary works much like BPE, by repeatedly merging pieces. Start with every character in the corpus as a token, marking characters that do not begin a word with a prefix; in BERT's convention, word becomes w ##o ##r ##d. Count how often each adjacent pair of tokens appears. Then merge the pair with the highest score and add the result to the vocabulary, repeating until the vocabulary reaches the target size. The difference from BPE is the score. BPE merges the most frequent pair. The WordPiece description in Hugging Face's course divides the pair's frequency by the product of the frequencies of its two parts, which favors pairs whose parts rarely appear apart. The course notes that Google never open-sourced its training code, so this formula is a reconstruction from the literature.

Applying a trained vocabulary to new text is simpler. BERT's tokenizer first normalizes text, converting whitespace and, for uncased models, lowercasing and stripping accents. It then splits off punctuation. Finally it runs WordPiece on each word with greedy longest-match-first: find the longest vocabulary entry that matches the start of the word, emit it, and continue on the rest of the word using pieces with the ## prefix. If at any point no piece matches, the whole word becomes the unknown token [UNK], rather than only the unmatched letters. Only the final vocabulary is stored, not an ordered list of merges, which is another difference from BPE.

BERT made WordPiece famous. The October 2018 paper by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova used WordPiece embeddings with a 30,000-token vocabulary, citing the 2016 translation paper, along with special tokens such as [CLS] at the start of each input, [SEP] between sentences and [MASK] for pretraining. The BERT repository's example shows the whole pipeline: John Johanson's house becomes john johan ##son ' s house. Multilingual BERT extended the idea to 104 languages with a shared vocabulary of 110,000 entries, sampling languages with exponential smoothing so that English was drawn about 100 times more often than Icelandic instead of about 1,000 times.

Compared with alternatives, WordPiece and BPE produce similar vocabularies and similar results; the 2016 translation paper reported that a 32,000-wordpiece model reached 38.95 BLEU on WMT English to French against 37.90 for a word model and 38.01 for a character model, and 24.61 against 23.12 and 22.62 on English to German. The main practical differences are the merge score, the longest-match inference and the whole-word [UNK] fallback. SentencePiece and byte-level BPE, used in many later models, work directly on raw text or bytes and so never need an unknown token, while WordPiece as used in BERT depends on a pre-tokenization step that splits on spaces and punctuation.

WordPiece has known weaknesses. Because it relies on whitespace and punctuation splitting, it needs special handling for languages without spaces; multilingual BERT puts spaces around every Chinese, Japanese Kanji and Korean Hanja character before WordPiece runs. Uncased models strip accents, which the multilingual BERT authors admit can remove meaning in some languages. A word with even one character missing from the vocabulary becomes [UNK] entirely. Subword splits also do not follow morphology: a meaningful unit like un or able may or may not survive as a piece, which the course illustrates by noting that a frequent pair can still score low. And token counts vary by language, so the same sentence can cost many more tokens in a language underrepresented in the vocabulary.

In practice, WordPiece is cheap and well supported. The original BERT checkpoints ship a vocab.txt file mapping each piece to an ID, and changing the vocabulary requires updating the model's vocab_size setting. Libraries such as Hugging Face Tokenizers and TensorFlow Text implement it, and Google's Fast WordPiece algorithm, presented at EMNLP 2021, made tokenization linear in input length. The key operational rule is to use exactly the tokenizer a model was trained with, including its casing setting, since a mismatch silently changes every token ID.