Nyelvi AI ÚTMUTATÓ

XLNet and Permutation Language Modeling

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token.

Hallgassa meg ezt az útmutatótAI-narrált · ~7 min

Ezen az oldalon8 perc olvasás
  1. Áttekintés
  2. Főbb tanulságok
  3. Mély merülés
  4. Tracing one permutation of a four-word sentence
  5. Valós megvalósítás
  6. The Future of XLNet and Permutation Language Modeling
  7. A gyakorlatba ültetés
  8. Források és további olvasmányok
  9. Folytassa a felfedezést
  10. Gyakran ismételt kérdések

Áttekintés

It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

Főbb tanulságok

  1. XLNet, published at NeurIPS 2019 by Carnegie Mellon University and Google Brain researchers, trains by predicting tokens in randomly sampled factorization orders.
  2. Permutation language modeling gives each token bidirectional context while staying autoregressive, and it never inserts a [MASK] token into the input.
  3. Two-stream self-attention separates a content stream that sees each token from a query stream that knows only the target position.
  4. XLNet reported 85.4% on RACE versus 72.0% for BERT, and beat BERT on 20 tasks under comparable settings.
  5. RoBERTa narrowed the gap within weeks, showing that much of BERT's shortfall came from undertraining rather than its masked objective alone.

Mély merülés

XLNet is a pretrained transformer language model introduced in June 2019 by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell and Ruslan Salakhutdinov of Carnegie Mellon University and Quoc V. Le of the Google Brain team, and published at NeurIPS 2019. Its key contribution is a training objective called permutation language modeling. The goal was to combine the strengths of two earlier families of models: autoregressive models such as GPT, which predict each word from the words before it, and autoencoding models such as BERT, which hide some words and predict them from both sides.

The problem XLNet set out to fix is worth stating plainly. An autoregressive model only sees context on one side, which limits understanding tasks where both sides matter. BERT sees both sides by replacing a share of input tokens with a special [MASK] symbol and predicting the originals. The XLNet authors identified two weaknesses in that design. First, the [MASK] symbol appears in pretraining but never in real downstream text, a mismatch they call the pretrain-finetune discrepancy. Second, BERT predicts the masked tokens independently of one another, so if both words of a name like New York are masked, it cannot use one guess to inform the other.

Permutation language modeling works like this. Take a sentence of T tokens and pick a random factorization order, a shuffled sequence of positions such as 3, 1, 4, 2. The model then predicts the tokens one at a time in that order, so token 3 is predicted from nothing, token 1 from token 3, token 4 from tokens 3 and 1, and so on. Crucially, the words themselves are never reordered in the input; their original positions are kept, and only the attention mask changes so that each token can see exactly the tokens that come earlier in the sampled order. Over many training steps, with a fresh order each time, every token ends up being predicted from many different combinations of left and right neighbors. The model is still autoregressive, multiplying conditional probabilities, but in expectation it learns from bidirectional context, and no [MASK] token is ever inserted.

This creates a technical puzzle that the paper solves with two-stream self-attention. To predict the token at a given position, the model must know which position it is predicting, or it would give the same answer for every target, but it must not see that token's own content, or the task would be trivial. XLNet therefore keeps two representations per position. The content stream works like a normal transformer and includes the token itself. The query stream knows the target position but not its word, and is the one used to make the prediction. Because predicting from very short contexts early in an order is hard and slows training, XLNet also uses partial prediction, predicting only the last part of each order, roughly one token in K, with K set to 6 for the large model.

XLNet also borrowed two components from Transformer-XL, a model the same group published at ACL 2019 that learned dependencies 450% longer than vanilla transformers. These were relative positional encodings, which describe how far apart two tokens are rather than their absolute index, and segment-level recurrence, which caches hidden states from a previous text segment as memory for the next. The large model used BERT-Large's architecture, 24 layers with a hidden size of 1,024 and 16 attention heads, and was pretrained on 32.89 billion subword pieces drawn from BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl. Training used 512 TPU v3 chips for 500,000 steps with a batch size of 8,192 and sequence length of 512, about 5.5 days.

The headline results were strong for mid-2019. The abstract reports XLNet beating BERT on 20 tasks under comparable settings. On RACE, a reading comprehension test drawn from English exams, it scored 85.4% against BERT's 72.0%. On SQuAD 2.0 it reached 90.6 F1 on the dev set against 81.77 for BERT. The paper also ran a fair comparison, training XLNet on the same Wikipedia and books data with BERT's hyperparameters, and XLNet still came out ahead on every dataset in that table. Its ablations showed that both the permutation objective and the Transformer-XL memory contributed, though some gaps were small: removing memory cut the base model's RACE score from 66.66 to 65.55.

Compared with the alternatives, the picture became more nuanced within weeks. RoBERTa, posted by Yinhan Liu and colleagues in July 2019, argued that the original BERT was significantly undertrained and showed that a better-tuned masked language model with more data could match or exceed later models. XLNet's own updated paper compares against RoBERTa and reports generally better scores, for example 90.8% versus 90.2% on MNLI, but the margins were far smaller than over the original BERT. This made it harder to say how much of XLNet's gain came from the permutation objective and how much from more data, longer training and Transformer-XL components.

XLNet's practical limitations were mostly about cost and complexity. Two-stream attention and permutation masks make the model harder to implement and slower to train than a standard masked model. The official repository warns that fine-tuning XLNet-Large on a 16 GB GPU fits only a single 512-token sequence per batch, and that reproducing many paper results would take roughly 32 to 128 GPUs. Partial prediction means each sequence trains on only a fraction of its tokens. XLNet is now studied mainly as a reference point: it remains a clear demonstration that an autoregressive model can learn bidirectional context without corrupting its input.

04Működő példa

Tracing one permutation of a four-word sentence

  1. Write the sentence 'cats chase small mice' with positions 1 to 4.

  2. Sample a factorization order, for example 3, 1, 4, 2.

  3. Predict position 3, 'small', with no visible tokens, using only the query stream's knowledge that position 3 is the target.

  4. Predict position 1, 'cats', seeing only 'small' at position 3.

  5. Predict position 4, 'mice', seeing 'small' and 'cats', both of which sit to its left in the text.

  6. Predict position 2, 'chase', seeing positions 1, 3 and 4, so it uses context from both its left and right.

  7. With partial prediction and K = 2, keep only the losses for the last two steps, positions 4 and 2.

Mit mutat

Position 2 was predicted from words on both sides without any [MASK] token, which is the core of permutation language modeling. Check the trace by building the 4 by 4 attention mask: row 2 should allow columns 1, 3 and 4, and row 3 should allow none.

Valós megvalósítás

Reading comprehension: on the RACE exam-question benchmark, the XLNet paper reports 85.4% accuracy, against 72.0% for BERT and 83.2% for RoBERTa.

Extractive question answering on SQuAD 2.0, where the paper reports a single-model dev score of 90.6 F1, compared with 81.77 for BERT and 89.4 for RoBERTa.

Fine-tuning released checkpoints: the authors' Apache-2.0 repository published XLNet-Large on June 19, 2019 and XLNet-Base on July 16, 2019 for classification, question answering and ranking tasks.

Teaching and research on pretraining objectives, where XLNet is the standard example of getting bidirectional context while keeping an autoregressive, left-to-right style probability model.

Natural language inference: on GLUE's MNLI task, the paper reports 90.8% dev accuracy, ahead of RoBERTa's 90.2% and BERT's 86.6%.

The Future of XLNet and Permutation Language Modeling

XLNet's direct line of development did not continue at large scale, but the questions it raised remain open. The rapid arrival of RoBERTa in July 2019, which matched or beat later models with a tuned masked objective, showed how hard it is to separate the effect of a pretraining objective from the effect of data, compute and training length. That lesson shaped how later papers report ablations and fair comparisons. The idea of learning from many orderings, rather than a single left-to-right order, continues to appear in research on non-autoregressive generation, infilling, and models that can generate tokens in flexible order. The XLNet paper itself, dated 2019 and revised in January 2020, framed permutation modeling as a general bridge between autoregressive and autoencoding pretraining. Whether flexible-order objectives offer advantages over plain next-token prediction at the scale of current large language models is an open research question, and claims either way depend on carefully controlled comparisons of the kind the XLNet and RoBERTa papers attempted.

A gyakorlatba ültetés

Miért számít

  • Sebesség és méretarány. A nyelvi munkafolyamatok gyorsabban haladhatnak a következetesség feláldozása nélkül.
  • Hozzáférés és elérés. Kibővíti a hozzáférést a nyelvek és a kommunikációs stílusok között.
  • Tisztább döntések. A csapatok több időt tölthetnek az ítélkezéssel, míg az automatizálás kezeli az ismétlést.

Vigyázz

  • A hallucinált tények csendben bekerülhetnek a jelentésekbe, a támogatási folyamatokba vagy a kutatási eredményekbe.
  • Az azonnali érzékenység inkonzisztens eredményeket eredményezhet a hasonló kérések között.
  • Ha a hozzáférés-szabályozás gyenge, az érzékeny szöveges adatok megjelenhetnek.

Következő lépések

  1. A kiadás előtt határozza meg a kimeneti formátumot, hangszínt és minőségi szabványokat.
  2. Földelje a válaszokat megbízható forrásokból, amikor a pontosság számít.
  3. Tartson emberi ellenőrzési pontot a nagy tétű kimenetekhez.
  4. Kövesse nyomon a meghibásodási mintákat, és rendszeresen tanítsa át az utasításokat vagy a munkafolyamatokat.

Források és további olvasmányok

  1. arXiv (Yang et al.)XLNet: Generalized Autoregressive Pretraining for Language Understanding
  2. NeurIPS 2019 ProceedingsXLNet: Generalized Autoregressive Pretraining for Language Understanding
  3. arXiv / ACL 2019 (Dai et al.)Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
  4. arXiv (Liu et al.)RoBERTa: A Robustly Optimized BERT Pretraining Approach
  5. GitHub (XLNet authors)zihangdai/xlnet

Folytassa a felfedezést

Ingyenes hírlevél

Szerezze meg a napi AI összefoglalót

Minden hétköznap reggel három ellenőrzött AI történet, egyszerű angol nyelven. Örökké ingyen, hirdetések nélkül.

Minden hétköznap egy e-mail. Egy kattintással lemondhat. Soha nem értékesítjük vagy osztjuk meg az Ön címét.

Teszteld magad

Take the XLNet and Permutation Language Modeling quiz

Azonnali visszajelzés minden válaszra, és megosztható tanúsítvány egy ellenőrizhető azonosítóval, miután sikeresen teljesít egy kurzust.

Kezdő kvíz

Támogassa az ingyenes AI oktatást. AI Understanding egy 501(c)(3) nonprofit – nincs hirdetés, nincs fizetőfal, soha. Tegyen adományt

Gyakran ismételt kérdések

What is XLNet and Permutation Language Modeling?

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token. It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

Milyen edzés előtti célt használ az XLNet?

Az XLNet maximalizálja a várható naplózási valószínűséget a jogkivonat faktorizációs sorrendjének összes permutációjában, amelyet permutációs nyelvi modellezésnek neveznek.

Melyik BERT gyengeség kezelésére tervezték az XLNetet?

A BERT mesterséges [MASK] szimbóluma soha nem jelenik meg a finomhangolás során, és a maszkolt előrejelzéseket függetlenként kezeli; Az XLNet mindkét problémát elkerüli.

Miben különbözik egymástól az XLNet kétfolyamos önfigyelése?

A tartalomfolyam kódolja a tokent és a kontextust, míg a lekérdezési adatfolyam ismeri a cél pozícióját, de a tartalmát nem, lehetővé téve a szivárgás nélküli előrejelzést.

Az XLNet fizikailag megkeveri a szavakat egy mondatban?

Az XLNet figyelemmaszkokon keresztül módosítja az előrejelzési (faktorizációs) sorrendet; az eredeti token pozíciókat pozíciókódolások őrzik meg.

Melyik architektúra szolgálja az XLNet gerincét a hosszú távú környezethez?

Az XLNet a Transformer-XL-re épít, amely szegmens-ismétlődést és relatív pozíciókódolást ad hozzá a hosszú szekvenciák kezeléséhez.