本頁8 分鐘閱讀
概述
Developed at Google for Japanese and Korean voice search in 2012, it became the tokenizer behind Google's neural translation system and BERT, and it shaped how a generation of language models reads text.
重點摘要
- WordPiece splits words into subword pieces from a fixed vocabulary, marking continuation pieces with ## in BERT's convention.
- It was introduced by Schuster and Nakajima at Google in 2012 for Japanese and Korean voice search.
- BERT, published in 2018, used WordPiece embeddings with a 30,000-token vocabulary.
- At inference WordPiece uses greedy longest-match-first, and a word with no matching piece becomes [UNK] entirely.
- Unlike BPE, WordPiece picks merges by a score that divides a pair's frequency by its parts' frequencies, not by raw frequency.
- Google's Fast WordPiece algorithm tokenizes in linear time and was reported 8.2 times faster than Hugging Face Tokenizers.
深入探討
Tokenization is the first step in any language model: turning raw text into a sequence of units, called tokens, that the model has embeddings for. Splitting on spaces gives whole words, but no fixed word list can cover every name, typo, technical term and inflected form, and every unknown word collapses into a single unknown token. Splitting into characters covers everything but makes sequences long and forces the model to learn spelling from scratch. Subword tokenization sits between the two: common words stay whole, and rare words are broken into smaller pieces that are in the vocabulary. WordPiece is one of the main subword methods, alongside byte-pair encoding (BPE) and the unigram language model used in SentencePiece.
WordPiece comes from Google. Mike Schuster and Kaisuke Nakajima described it in a 2012 ICASSP paper on Japanese and Korean voice search, where text has no spaces between words and a large character set, so a speech recognizer needed a data-driven way to choose units. Google's 2016 neural machine translation paper by Yonghui Wu and colleagues adopted the same model and explained its goal: given a training corpus and a target number of tokens D, pick D wordpieces so the corpus can be segmented into as few pieces as possible, which amounts to maximizing the likelihood of the data under a simple language model. Its example sentence segments Jet makers feud over seat width as _J et _makers _fe ud _over _seat _width, where the underscore marks the start of a word and common words stay whole.
Training a WordPiece vocabulary works much like BPE, by repeatedly merging pieces. Start with every character in the corpus as a token, marking characters that do not begin a word with a prefix; in BERT's convention, word becomes w ##o ##r ##d. Count how often each adjacent pair of tokens appears. Then merge the pair with the highest score and add the result to the vocabulary, repeating until the vocabulary reaches the target size. The difference from BPE is the score. BPE merges the most frequent pair. The WordPiece description in Hugging Face's course divides the pair's frequency by the product of the frequencies of its two parts, which favors pairs whose parts rarely appear apart. The course notes that Google never open-sourced its training code, so this formula is a reconstruction from the literature.
Applying a trained vocabulary to new text is simpler. BERT's tokenizer first normalizes text, converting whitespace and, for uncased models, lowercasing and stripping accents. It then splits off punctuation. Finally it runs WordPiece on each word with greedy longest-match-first: find the longest vocabulary entry that matches the start of the word, emit it, and continue on the rest of the word using pieces with the ## prefix. If at any point no piece matches, the whole word becomes the unknown token [UNK], rather than only the unmatched letters. Only the final vocabulary is stored, not an ordered list of merges, which is another difference from BPE.
BERT made WordPiece famous. The October 2018 paper by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova used WordPiece embeddings with a 30,000-token vocabulary, citing the 2016 translation paper, along with special tokens such as [CLS] at the start of each input, [SEP] between sentences and [MASK] for pretraining. The BERT repository's example shows the whole pipeline: John Johanson's house becomes john johan ##son ' s house. Multilingual BERT extended the idea to 104 languages with a shared vocabulary of 110,000 entries, sampling languages with exponential smoothing so that English was drawn about 100 times more often than Icelandic instead of about 1,000 times.
Compared with alternatives, WordPiece and BPE produce similar vocabularies and similar results; the 2016 translation paper reported that a 32,000-wordpiece model reached 38.95 BLEU on WMT English to French against 37.90 for a word model and 38.01 for a character model, and 24.61 against 23.12 and 22.62 on English to German. The main practical differences are the merge score, the longest-match inference and the whole-word [UNK] fallback. SentencePiece and byte-level BPE, used in many later models, work directly on raw text or bytes and so never need an unknown token, while WordPiece as used in BERT depends on a pre-tokenization step that splits on spaces and punctuation.
WordPiece has known weaknesses. Because it relies on whitespace and punctuation splitting, it needs special handling for languages without spaces; multilingual BERT puts spaces around every Chinese, Japanese Kanji and Korean Hanja character before WordPiece runs. Uncased models strip accents, which the multilingual BERT authors admit can remove meaning in some languages. A word with even one character missing from the vocabulary becomes [UNK] entirely. Subword splits also do not follow morphology: a meaningful unit like un or able may or may not survive as a piece, which the course illustrates by noting that a frequent pair can still score low. And token counts vary by language, so the same sentence can cost many more tokens in a language underrepresented in the vocabulary.
In practice, WordPiece is cheap and well supported. The original BERT checkpoints ship a vocab.txt file mapping each piece to an ID, and changing the vocabulary requires updating the model's vocab_size setting. Libraries such as Hugging Face Tokenizers and TensorFlow Text implement it, and Google's Fast WordPiece algorithm, presented at EMNLP 2021, made tokenization linear in input length. The key operational rule is to use exactly the tokenizer a model was trained with, including its casing setting, since a mismatch silently changes every token ID.
04工作範例
Tokenizing a sentence by hand with greedy longest match
Take a toy vocabulary: john, johan, ##son, ', s, house, h, ##o, ##u, ##s, ##e and [UNK].
Normalize the sentence John Johanson's house to lowercase and split off punctuation to get john johanson ' s house.
For john, the longest match from the start is john itself, so emit john.
For johanson, the longest match is johan; continue on son with the prefix and match ##son.
Emit ' and s as single pieces, then match house whole.
Now try the word hoax: h matches, then ##o, but no piece starts with ##a, so the entire word becomes [UNK].
它顯示了什麼
The sentence tokenizes to john johan ##son ' s house, matching the example in the BERT repository. The hoax case shows the whole-word [UNK] fallback, which you can confirm by running a BERT tokenizer library on the same text.
現實世界的實施
Preparing input for BERT, which the 2018 paper says uses WordPiece embeddings with a 30,000-token vocabulary, so John Johanson's house becomes john johan ##son ' s house in the uncased model.
Labeling named entities with BERT by classifying only the first sub-token of each word, the approach the BERT paper used on CoNLL-2003, and mapping labels back with a word-to-token index map.
Serving multilingual text with multilingual BERT, whose cased model covers 104 languages with one shared 110,000-entry WordPiece vocabulary and splits Chinese characters before WordPiece runs.
Speeding up tokenization in production with Google's Fast WordPiece algorithm, which a 2021 EMNLP paper reports is on average 8.2 times faster than Hugging Face Tokenizers on general text.
Choosing a vocabulary size for a translation model, guided by Google's 2016 neural machine translation paper, which found 8,000 to 32,000 wordpieces gave good BLEU scores and fast decoding.
WordPiece 代幣化的未來
WordPiece's influence now runs mostly through the large installed base of BERT-family encoders used for search, classification and named entity recognition, while many newer generative models use byte-level BPE or SentencePiece tokenizers that avoid unknown tokens. Engineering work on WordPiece itself has focused on speed, as in Google's 2021 Fast WordPiece algorithm with linear-time tokenization. Open problems are shared with all subword schemes. Multilingual fairness is one: multilingual BERT's documentation shows how much vocabulary allocation depends on how languages are sampled, and underrepresented languages end up split into more, shorter pieces. Robustness is another, since a single unusual character can turn a whole word into [UNK]. Whether future models keep fixed subword vocabularies or move toward byte- or character-level inputs with learned grouping is an open research question rather than a settled trend.
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
資料來源與延伸閱讀
- Google Research (ICASSP 2012)Japanese and Korean Voice Search
- arXivGoogle's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- arXiv (NAACL 2019)BERT:用於語言理解的深度雙向變壓器的預訓練
- GitHub (Google Research)google-research/bert
- GitHub (Google Research)Multilingual BERT
- Hugging Face LLM CourseWordPiece tokenization
- arXiv (EMNLP 2021)Fast WordPiece Tokenization
不斷探索
常見問題
什麼是 WordPiece 標記化?
WordPiece is a subword tokenization method that splits text into pieces from a fixed vocabulary, so rare words like johanson become known parts such as johan and ##son. Developed at Google for Japanese and Korean voice search in 2012, it became the tokenizer behind Google's neural translation system and BERT, and it shaped how a generation of language models reads text.
WordPiece 輸出中的「##」前綴表示什麼?
WordPiece 用“##”標記子字延續,因此“playing”變成“play”+“##ing”,讓解碼器重建字邊界。
原始 BERT 使用的 WordPiece 詞彙量大約有多大?
BERT 使用大約 30,000 個子字單元的 WordPiece 詞彙表,平衡覆蓋範圍與嵌入表大小。
WordPiece 的合併標準與標準位元組對編碼有何不同?
BPE 合併最頻繁的相鄰對,而 WordPiece 合併最能增加語料庫可能性的對,優先考慮聯合頻率超過其部分乘積的對。
如果 BERT 風格的 WordPiece 分詞器無法將單字切分為詞彙片段,它會傳回什麼?
WordPiece 分詞器使用已知的詞彙片段,如果在找不到有效切分的情況下可以傳回未知 token 標記。字元後備機制並不保證適用於所有分詞器設定。
在推理時,WordPiece如何選擇如何分割單字?
WordPiece 貪婪地進行標記,從目前位置開始匹配最長的詞彙條目,然後重複其餘部分。