Byte-Pair Encoding

BPE was developed for data compression and adapted to word segmentation in the translation research by Sennrich, Haddow, and Birch, published in 2016. In a common subword-training procedure, text is represented through initial symbols such as characters. Adjacent pairs are counted over the training representation, and a frequent pair is merged into a larger symbol. Repeating the procedure learns reusable pieces. More frequent sequences can gain larger units, while an unfamiliar word may remain split into smaller ones. The precise behavior depends on the training data and implementation; not every common word must become one token. Training produces a vocabulary and an ordered collection of merge rules. GPT-2's published encoder applies eligible pairs according to learned ranks rather than recounting the new input to invent a new vocabulary. It first represents UTF-8 bytes through a reversible byte-to-symbol mapping. The byte alphabet covers 256 values, permitting valid UTF-8 input to be represented through its constituent bytes when larger merged pieces are unavailable. This avoids the character-alphabet coverage limitation under that configuration, but encoding success does not establish that the model knows the text's meaning. Vocabulary accounting can be explicit. In an illustrative configuration with 256 base byte tokens, two distinct additional merge tokens, and one separate special token, the total is 259. Decoding has another important order: GPT-2's implementation joins decoded token pieces, reconstructs the complete byte sequence, then decodes that sequence as UTF-8. Individual pieces need not each correspond to an independently complete Unicode character.