語言人工智慧指南

XLNet and Permutation Language Modeling

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token.

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Tracing one permutation of a four-word sentence
  5. 現實世界的實施
  6. The Future of XLNet and Permutation Language Modeling
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

重點摘要

  1. XLNet, published at NeurIPS 2019 by Carnegie Mellon University and Google Brain researchers, trains by predicting tokens in randomly sampled factorization orders.
  2. Permutation language modeling gives each token bidirectional context while staying autoregressive, and it never inserts a [MASK] token into the input.
  3. Two-stream self-attention separates a content stream that sees each token from a query stream that knows only the target position.
  4. XLNet reported 85.4% on RACE versus 72.0% for BERT, and beat BERT on 20 tasks under comparable settings.
  5. RoBERTa narrowed the gap within weeks, showing that much of BERT's shortfall came from undertraining rather than its masked objective alone.

深入探討

XLNet is a pretrained transformer language model introduced in June 2019 by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell and Ruslan Salakhutdinov of Carnegie Mellon University and Quoc V. Le of the Google Brain team, and published at NeurIPS 2019. Its key contribution is a training objective called permutation language modeling. The goal was to combine the strengths of two earlier families of models: autoregressive models such as GPT, which predict each word from the words before it, and autoencoding models such as BERT, which hide some words and predict them from both sides.

The problem XLNet set out to fix is worth stating plainly. An autoregressive model only sees context on one side, which limits understanding tasks where both sides matter. BERT sees both sides by replacing a share of input tokens with a special [MASK] symbol and predicting the originals. The XLNet authors identified two weaknesses in that design. First, the [MASK] symbol appears in pretraining but never in real downstream text, a mismatch they call the pretrain-finetune discrepancy. Second, BERT predicts the masked tokens independently of one another, so if both words of a name like New York are masked, it cannot use one guess to inform the other.

Permutation language modeling works like this. Take a sentence of T tokens and pick a random factorization order, a shuffled sequence of positions such as 3, 1, 4, 2. The model then predicts the tokens one at a time in that order, so token 3 is predicted from nothing, token 1 from token 3, token 4 from tokens 3 and 1, and so on. Crucially, the words themselves are never reordered in the input; their original positions are kept, and only the attention mask changes so that each token can see exactly the tokens that come earlier in the sampled order. Over many training steps, with a fresh order each time, every token ends up being predicted from many different combinations of left and right neighbors. The model is still autoregressive, multiplying conditional probabilities, but in expectation it learns from bidirectional context, and no [MASK] token is ever inserted.

This creates a technical puzzle that the paper solves with two-stream self-attention. To predict the token at a given position, the model must know which position it is predicting, or it would give the same answer for every target, but it must not see that token's own content, or the task would be trivial. XLNet therefore keeps two representations per position. The content stream works like a normal transformer and includes the token itself. The query stream knows the target position but not its word, and is the one used to make the prediction. Because predicting from very short contexts early in an order is hard and slows training, XLNet also uses partial prediction, predicting only the last part of each order, roughly one token in K, with K set to 6 for the large model.

XLNet also borrowed two components from Transformer-XL, a model the same group published at ACL 2019 that learned dependencies 450% longer than vanilla transformers. These were relative positional encodings, which describe how far apart two tokens are rather than their absolute index, and segment-level recurrence, which caches hidden states from a previous text segment as memory for the next. The large model used BERT-Large's architecture, 24 layers with a hidden size of 1,024 and 16 attention heads, and was pretrained on 32.89 billion subword pieces drawn from BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl. Training used 512 TPU v3 chips for 500,000 steps with a batch size of 8,192 and sequence length of 512, about 5.5 days.

The headline results were strong for mid-2019. The abstract reports XLNet beating BERT on 20 tasks under comparable settings. On RACE, a reading comprehension test drawn from English exams, it scored 85.4% against BERT's 72.0%. On SQuAD 2.0 it reached 90.6 F1 on the dev set against 81.77 for BERT. The paper also ran a fair comparison, training XLNet on the same Wikipedia and books data with BERT's hyperparameters, and XLNet still came out ahead on every dataset in that table. Its ablations showed that both the permutation objective and the Transformer-XL memory contributed, though some gaps were small: removing memory cut the base model's RACE score from 66.66 to 65.55.

Compared with the alternatives, the picture became more nuanced within weeks. RoBERTa, posted by Yinhan Liu and colleagues in July 2019, argued that the original BERT was significantly undertrained and showed that a better-tuned masked language model with more data could match or exceed later models. XLNet's own updated paper compares against RoBERTa and reports generally better scores, for example 90.8% versus 90.2% on MNLI, but the margins were far smaller than over the original BERT. This made it harder to say how much of XLNet's gain came from the permutation objective and how much from more data, longer training and Transformer-XL components.

XLNet's practical limitations were mostly about cost and complexity. Two-stream attention and permutation masks make the model harder to implement and slower to train than a standard masked model. The official repository warns that fine-tuning XLNet-Large on a 16 GB GPU fits only a single 512-token sequence per batch, and that reproducing many paper results would take roughly 32 to 128 GPUs. Partial prediction means each sequence trains on only a fraction of its tokens. XLNet is now studied mainly as a reference point: it remains a clear demonstration that an autoregressive model can learn bidirectional context without corrupting its input.

04工作範例

Tracing one permutation of a four-word sentence

  1. Write the sentence 'cats chase small mice' with positions 1 to 4.

  2. Sample a factorization order, for example 3, 1, 4, 2.

  3. Predict position 3, 'small', with no visible tokens, using only the query stream's knowledge that position 3 is the target.

  4. Predict position 1, 'cats', seeing only 'small' at position 3.

  5. Predict position 4, 'mice', seeing 'small' and 'cats', both of which sit to its left in the text.

  6. Predict position 2, 'chase', seeing positions 1, 3 and 4, so it uses context from both its left and right.

  7. With partial prediction and K = 2, keep only the losses for the last two steps, positions 4 and 2.

它顯示了什麼

Position 2 was predicted from words on both sides without any [MASK] token, which is the core of permutation language modeling. Check the trace by building the 4 by 4 attention mask: row 2 should allow columns 1, 3 and 4, and row 3 should allow none.

現實世界的實施

Reading comprehension: on the RACE exam-question benchmark, the XLNet paper reports 85.4% accuracy, against 72.0% for BERT and 83.2% for RoBERTa.

Extractive question answering on SQuAD 2.0, where the paper reports a single-model dev score of 90.6 F1, compared with 81.77 for BERT and 89.4 for RoBERTa.

Fine-tuning released checkpoints: the authors' Apache-2.0 repository published XLNet-Large on June 19, 2019 and XLNet-Base on July 16, 2019 for classification, question answering and ranking tasks.

Teaching and research on pretraining objectives, where XLNet is the standard example of getting bidirectional context while keeping an autoregressive, left-to-right style probability model.

Natural language inference: on GLUE's MNLI task, the paper reports 90.8% dev accuracy, ahead of RoBERTa's 90.2% and BERT's 86.6%.

The Future of XLNet and Permutation Language Modeling

XLNet's direct line of development did not continue at large scale, but the questions it raised remain open. The rapid arrival of RoBERTa in July 2019, which matched or beat later models with a tuned masked objective, showed how hard it is to separate the effect of a pretraining objective from the effect of data, compute and training length. That lesson shaped how later papers report ablations and fair comparisons. The idea of learning from many orderings, rather than a single left-to-right order, continues to appear in research on non-autoregressive generation, infilling, and models that can generate tokens in flexible order. The XLNet paper itself, dated 2019 and revised in January 2020, framed permutation modeling as a general bridge between autoregressive and autoencoding pretraining. Whether flexible-order objectives offer advantages over plain next-token prediction at the scale of current large language models is an open research question, and claims either way depend on carefully controlled comparisons of the kind the XLNet and RoBERTa papers attempted.

付諸實踐

為什麼這很重要

  • 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
  • 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
  • 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。

留意

  • 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
  • 及時的敏感性可能會在類似的請求中產生不一致的結果。
  • 如果存取控制薄弱,敏感文字資料可能會暴露。

後續步驟

  1. 在推出之前定義輸出格式、語氣和品質標準。
  2. 當準確性很重要時,請使用可信任來源進行地面回應。
  3. 為高風險輸出保留人工審查檢查點。
  4. 追蹤故障模式並定期重新訓練提示或工作流程。

資料來源與延伸閱讀

  1. arXiv (Yang et al.)XLNet: Generalized Autoregressive Pretraining for Language Understanding
  2. NeurIPS 2019 ProceedingsXLNet: Generalized Autoregressive Pretraining for Language Understanding
  3. arXiv / ACL 2019 (Dai et al.)Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
  4. arXiv (Liu et al.)RoBERTa: A Robustly Optimized BERT Pretraining Approach
  5. GitHub (XLNet authors)zihangdai/xlnet

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

Take the XLNet and Permutation Language Modeling quiz

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

What is XLNet and Permutation Language Modeling?

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token. It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

XLNet 使用什麼預訓練目標?

XLNet 最大化令牌分解順序的所有排列的預期對數似然,稱為排列語言建模。

XLNet 是專門為了解決哪些 BERT 弱點而設計的?

BERT 的人工 [MASK] 符號在微調期間不會出現,並且它將屏蔽預測視為獨立的; XLNet 避免了這兩個問題。

XLNet的兩流self-attention分離了什麼?

內容流對令牌和上下文進行編碼,而查詢流知道目標的位置但不知道其內容,從而實現無洩漏的預測。

XLNet 是否會物理地打亂句子中的單字?

XLNet 透過注意力遮罩排列預測(分解)順序;原始標記位置透過位置編碼保留。

哪種架構作為 XLNet 的遠端上下文骨幹網路?

XLNet 建立在 Transformer-XL 的基礎上,它添加了分段循環和相對位置編碼來處理長序列。