Language AI GUIDE
WordPiece Tokenization
WordPiece is a subword tokenization method used by BERT-family models.
Listen to this guideAI-narrated · ~1 min
On this page1 min read
Overview
It breaks text into vocabulary pieces, marking continuation pieces with a prefix such as ## in the original BERT tokenizer.
Deep Dive
The BERT paper describes a WordPiece vocabulary of about 30,000 tokens; multilingual BERT uses a shared vocabulary across languages. At inference, the tokenizer greedily selects the longest available piece, using ## for continuation pieces. If a token cannot be segmented under that tokenizer’s rules, it may emit [UNK]; character-level decomposition is not a universal fallback. WordPiece’s training criterion and behavior should be attributed to the specified tokenizer/model rather than every search engine or specialized BERT derivative.
Real-World Implementation
Original BERT uses a WordPiece vocabulary of about 30,000 tokens.
The ## prefix marks a continuation piece in BERT token output.
Multilingual BERT uses a shared WordPiece vocabulary across its training languages.
A tokenizer can emit [UNK] when no permitted vocabulary segmentation is available.
The Future of WordPiece Tokenization
Model-specific tokenizers should be documented with the model and vocabulary that use them. The cited sources do not establish a universal character-fallback behavior or prevalence across Google Search, medical BERT variants and future systems.
Putting it into practice
Why it matters
- Speed and scale. Language workflows can move faster without sacrificing consistency.
- Access and reach. It expands access across languages and communication styles.
- Clearer decisions. Teams can spend more time on judgment while automation handles repetition.
Watch out for
- Hallucinated facts can quietly enter reports, support flows, or research outputs.
- Prompt sensitivity can create inconsistent results across similar requests.
- Sensitive text data may be exposed if access controls are weak.
Next steps
- Define output format, tone, and quality standards before rollout.
- Ground responses with trusted sources whenever accuracy matters.
- Keep a human review checkpoint for high-stakes outputs.
- Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Keep up with AI in 3 minutes a day
One short email each weekday with the three AI stories that actually matter. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the WordPiece Tokenization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Frequently asked questions
What is WordPiece Tokenization?
WordPiece is a subword tokenization method used by BERT-family models. It breaks text into vocabulary pieces, marking continuation pieces with a prefix such as ## in the original BERT tokenizer.
What does the '##' prefix indicate in WordPiece output?
WordPiece marks subword continuations with '##', so 'playing' becomes 'play' + '##ing', letting the decoder reconstruct word boundaries.
Roughly how large is the WordPiece vocabulary used by the original BERT?
BERT uses a WordPiece vocabulary of roughly 30,000 subword units, balancing coverage against embedding-table size.
How does WordPiece's merge criterion differ from standard Byte-Pair Encoding?
BPE merges the most frequent adjacent pair, while WordPiece merges the pair that most increases corpus likelihood, favoring pairs whose joint frequency exceeds the product of their parts.
What does a BERT-style WordPiece tokenizer return if it cannot segment a word into vocabulary pieces?
WordPiece tokenizers use known vocabulary pieces and can return an unknown-token marker if no valid segmentation is found. Character fallback is not guaranteed for every tokenizer configuration.
At inference time, how does WordPiece choose how to split a word?
WordPiece tokenizes greedily, matching the longest vocabulary entry starting at the current position, then repeating on the remainder.
Keep learning
Related guides
More guides picked for this topic