AI Glossary Term
What does “BPE (Byte Pair Encoding)” mean?
Definition
A subword tokenization algorithm that merges the most frequent character pairs into reusable tokens.
How BPE (Byte Pair Encoding) works
Example
In a training corpus where 'low' and 'lower' are common, the pair 'l','o' merges first, then 'lo','w', and so on. Common words become single tokens, and rare words are broken into familiar pieces.
Why it matters
BPE balances vocabulary size against sequence length and avoids unknown words. Most large language models use a variant of it.
Common questions
Why subwords?
They handle rare and new words by splitting them into known parts.
Does BPE work on bytes?
Byte-level BPE starts from bytes, so any text can be represented.
Where did it come from?
It began as a data-compression method and was adapted for translation models in 2015.