XLNet and Permutation Language Modeling

XLNet is a pretrained transformer language model introduced in June 2019 by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell and Ruslan Salakhutdinov of Carnegie Mellon University and Quoc V. Le of the Google Brain team, and published at NeurIPS 2019. Its key contribution is a training objective called permutation language modeling. The goal was to combine the strengths of two earlier families of models: autoregressive models such as GPT, which predict each word from the words before it, and autoencoding models such as BERT, which hide some words and predict them from both sides.

The problem XLNet set out to fix is worth stating plainly. An autoregressive model only sees context on one side, which limits understanding tasks where both sides matter. BERT sees both sides by replacing a share of input tokens with a special [MASK] symbol and predicting the originals. The XLNet authors identified two weaknesses in that design. First, the [MASK] symbol appears in pretraining but never in real downstream text, a mismatch they call the pretrain-finetune discrepancy. Second, BERT predicts the masked tokens independently of one another, so if both words of a name like New York are masked, it cannot use one guess to inform the other.

Permutation language modeling works like this. Take a sentence of T tokens and pick a random factorization order, a shuffled sequence of positions such as 3, 1, 4, 2. The model then predicts the tokens one at a time in that order, so token 3 is predicted from nothing, token 1 from token 3, token 4 from tokens 3 and 1, and so on. Crucially, the words themselves are never reordered in the input; their original positions are kept, and only the attention mask changes so that each token can see exactly the tokens that come earlier in the sampled order. Over many training steps, with a fresh order each time, every token ends up being predicted from many different combinations of left and right neighbors. The model is still autoregressive, multiplying conditional probabilities, but in expectation it learns from bidirectional context, and no [MASK] token is ever inserted.

This creates a technical puzzle that the paper solves with two-stream self-attention. To predict the token at a given position, the model must know which position it is predicting, or it would give the same answer for every target, but it must not see that token's own content, or the task would be trivial. XLNet therefore keeps two representations per position. The content stream works like a normal transformer and includes the token itself. The query stream knows the target position but not its word, and is the one used to make the prediction. Because predicting from very short contexts early in an order is hard and slows training, XLNet also uses partial prediction, predicting only the last part of each order, roughly one token in K, with K set to 6 for the large model.

XLNet also borrowed two components from Transformer-XL, a model the same group published at ACL 2019 that learned dependencies 450% longer than vanilla transformers. These were relative positional encodings, which describe how far apart two tokens are rather than their absolute index, and segment-level recurrence, which caches hidden states from a previous text segment as memory for the next. The large model used BERT-Large's architecture, 24 layers with a hidden size of 1,024 and 16 attention heads, and was pretrained on 32.89 billion subword pieces drawn from BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl. Training used 512 TPU v3 chips for 500,000 steps with a batch size of 8,192 and sequence length of 512, about 5.5 days.

The headline results were strong for mid-2019. The abstract reports XLNet beating BERT on 20 tasks under comparable settings. On RACE, a reading comprehension test drawn from English exams, it scored 85.4% against BERT's 72.0%. On SQuAD 2.0 it reached 90.6 F1 on the dev set against 81.77 for BERT. The paper also ran a fair comparison, training XLNet on the same Wikipedia and books data with BERT's hyperparameters, and XLNet still came out ahead on every dataset in that table. Its ablations showed that both the permutation objective and the Transformer-XL memory contributed, though some gaps were small: removing memory cut the base model's RACE score from 66.66 to 65.55.

Compared with the alternatives, the picture became more nuanced within weeks. RoBERTa, posted by Yinhan Liu and colleagues in July 2019, argued that the original BERT was significantly undertrained and showed that a better-tuned masked language model with more data could match or exceed later models. XLNet's own updated paper compares against RoBERTa and reports generally better scores, for example 90.8% versus 90.2% on MNLI, but the margins were far smaller than over the original BERT. This made it harder to say how much of XLNet's gain came from the permutation objective and how much from more data, longer training and Transformer-XL components.

XLNet's practical limitations were mostly about cost and complexity. Two-stream attention and permutation masks make the model harder to implement and slower to train than a standard masked model. The official repository warns that fine-tuning XLNet-Large on a 16 GB GPU fits only a single 512-token sequence per batch, and that reproducing many paper results would take roughly 32 to 128 GPUs. Partial prediction means each sequence trains on only a fraction of its tokens. XLNet is now studied mainly as a reference point: it remains a clear demonstration that an autoregressive model can learn bidirectional context without corrupting its input.