Ururimi AI

XLNet and Permutation Language Modeling

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token.

Umva iki gitaboAI-byavuzwe · ~7 min

Kuriyi page8 min soma
  1. Incamake
  2. Ibyingenzi byingenzi
  3. Kwibira cyane
  4. Tracing one permutation of a four-word sentence
  5. Gushyira mu bikorwa Isi
  6. The Future of XLNet and Permutation Language Modeling
  7. Kubishyira mubikorwa
  8. Inkomoko no gusoma
  9. Komeza Ubushakashatsi
  10. Ibibazo bikunze kubazwa

Incamake

It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

Ibyingenzi byingenzi

  1. XLNet, published at NeurIPS 2019 by Carnegie Mellon University and Google Brain researchers, trains by predicting tokens in randomly sampled factorization orders.
  2. Permutation language modeling gives each token bidirectional context while staying autoregressive, and it never inserts a [MASK] token into the input.
  3. Two-stream self-attention separates a content stream that sees each token from a query stream that knows only the target position.
  4. XLNet reported 85.4% on RACE versus 72.0% for BERT, and beat BERT on 20 tasks under comparable settings.
  5. RoBERTa narrowed the gap within weeks, showing that much of BERT's shortfall came from undertraining rather than its masked objective alone.

Kwibira cyane

XLNet is a pretrained transformer language model introduced in June 2019 by Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell and Ruslan Salakhutdinov of Carnegie Mellon University and Quoc V. Le of the Google Brain team, and published at NeurIPS 2019. Its key contribution is a training objective called permutation language modeling. The goal was to combine the strengths of two earlier families of models: autoregressive models such as GPT, which predict each word from the words before it, and autoencoding models such as BERT, which hide some words and predict them from both sides.

The problem XLNet set out to fix is worth stating plainly. An autoregressive model only sees context on one side, which limits understanding tasks where both sides matter. BERT sees both sides by replacing a share of input tokens with a special [MASK] symbol and predicting the originals. The XLNet authors identified two weaknesses in that design. First, the [MASK] symbol appears in pretraining but never in real downstream text, a mismatch they call the pretrain-finetune discrepancy. Second, BERT predicts the masked tokens independently of one another, so if both words of a name like New York are masked, it cannot use one guess to inform the other.

Permutation language modeling works like this. Take a sentence of T tokens and pick a random factorization order, a shuffled sequence of positions such as 3, 1, 4, 2. The model then predicts the tokens one at a time in that order, so token 3 is predicted from nothing, token 1 from token 3, token 4 from tokens 3 and 1, and so on. Crucially, the words themselves are never reordered in the input; their original positions are kept, and only the attention mask changes so that each token can see exactly the tokens that come earlier in the sampled order. Over many training steps, with a fresh order each time, every token ends up being predicted from many different combinations of left and right neighbors. The model is still autoregressive, multiplying conditional probabilities, but in expectation it learns from bidirectional context, and no [MASK] token is ever inserted.

This creates a technical puzzle that the paper solves with two-stream self-attention. To predict the token at a given position, the model must know which position it is predicting, or it would give the same answer for every target, but it must not see that token's own content, or the task would be trivial. XLNet therefore keeps two representations per position. The content stream works like a normal transformer and includes the token itself. The query stream knows the target position but not its word, and is the one used to make the prediction. Because predicting from very short contexts early in an order is hard and slows training, XLNet also uses partial prediction, predicting only the last part of each order, roughly one token in K, with K set to 6 for the large model.

XLNet also borrowed two components from Transformer-XL, a model the same group published at ACL 2019 that learned dependencies 450% longer than vanilla transformers. These were relative positional encodings, which describe how far apart two tokens are rather than their absolute index, and segment-level recurrence, which caches hidden states from a previous text segment as memory for the next. The large model used BERT-Large's architecture, 24 layers with a hidden size of 1,024 and 16 attention heads, and was pretrained on 32.89 billion subword pieces drawn from BooksCorpus, English Wikipedia, Giga5, ClueWeb 2012-B and Common Crawl. Training used 512 TPU v3 chips for 500,000 steps with a batch size of 8,192 and sequence length of 512, about 5.5 days.

The headline results were strong for mid-2019. The abstract reports XLNet beating BERT on 20 tasks under comparable settings. On RACE, a reading comprehension test drawn from English exams, it scored 85.4% against BERT's 72.0%. On SQuAD 2.0 it reached 90.6 F1 on the dev set against 81.77 for BERT. The paper also ran a fair comparison, training XLNet on the same Wikipedia and books data with BERT's hyperparameters, and XLNet still came out ahead on every dataset in that table. Its ablations showed that both the permutation objective and the Transformer-XL memory contributed, though some gaps were small: removing memory cut the base model's RACE score from 66.66 to 65.55.

Compared with the alternatives, the picture became more nuanced within weeks. RoBERTa, posted by Yinhan Liu and colleagues in July 2019, argued that the original BERT was significantly undertrained and showed that a better-tuned masked language model with more data could match or exceed later models. XLNet's own updated paper compares against RoBERTa and reports generally better scores, for example 90.8% versus 90.2% on MNLI, but the margins were far smaller than over the original BERT. This made it harder to say how much of XLNet's gain came from the permutation objective and how much from more data, longer training and Transformer-XL components.

XLNet's practical limitations were mostly about cost and complexity. Two-stream attention and permutation masks make the model harder to implement and slower to train than a standard masked model. The official repository warns that fine-tuning XLNet-Large on a 16 GB GPU fits only a single 512-token sequence per batch, and that reproducing many paper results would take roughly 32 to 128 GPUs. Partial prediction means each sequence trains on only a fraction of its tokens. XLNet is now studied mainly as a reference point: it remains a clear demonstration that an autoregressive model can learn bidirectional context without corrupting its input.

04Urugero rwakazi

Tracing one permutation of a four-word sentence

  1. Write the sentence 'cats chase small mice' with positions 1 to 4.

  2. Sample a factorization order, for example 3, 1, 4, 2.

  3. Predict position 3, 'small', with no visible tokens, using only the query stream's knowledge that position 3 is the target.

  4. Predict position 1, 'cats', seeing only 'small' at position 3.

  5. Predict position 4, 'mice', seeing 'small' and 'cats', both of which sit to its left in the text.

  6. Predict position 2, 'chase', seeing positions 1, 3 and 4, so it uses context from both its left and right.

  7. With partial prediction and K = 2, keep only the losses for the last two steps, positions 4 and 2.

Ibyo byerekana

Position 2 was predicted from words on both sides without any [MASK] token, which is the core of permutation language modeling. Check the trace by building the 4 by 4 attention mask: row 2 should allow columns 1, 3 and 4, and row 3 should allow none.

Gushyira mu bikorwa Isi

Reading comprehension: on the RACE exam-question benchmark, the XLNet paper reports 85.4% accuracy, against 72.0% for BERT and 83.2% for RoBERTa.

Extractive question answering on SQuAD 2.0, where the paper reports a single-model dev score of 90.6 F1, compared with 81.77 for BERT and 89.4 for RoBERTa.

Fine-tuning released checkpoints: the authors' Apache-2.0 repository published XLNet-Large on June 19, 2019 and XLNet-Base on July 16, 2019 for classification, question answering and ranking tasks.

Teaching and research on pretraining objectives, where XLNet is the standard example of getting bidirectional context while keeping an autoregressive, left-to-right style probability model.

Natural language inference: on GLUE's MNLI task, the paper reports 90.8% dev accuracy, ahead of RoBERTa's 90.2% and BERT's 86.6%.

The Future of XLNet and Permutation Language Modeling

XLNet's direct line of development did not continue at large scale, but the questions it raised remain open. The rapid arrival of RoBERTa in July 2019, which matched or beat later models with a tuned masked objective, showed how hard it is to separate the effect of a pretraining objective from the effect of data, compute and training length. That lesson shaped how later papers report ablations and fair comparisons. The idea of learning from many orderings, rather than a single left-to-right order, continues to appear in research on non-autoregressive generation, infilling, and models that can generate tokens in flexible order. The XLNet paper itself, dated 2019 and revised in January 2020, framed permutation modeling as a general bridge between autoregressive and autoencoding pretraining. Whether flexible-order objectives offer advantages over plain next-token prediction at the scale of current large language models is an open research question, and claims either way depend on carefully controlled comparisons of the kind the XLNet and RoBERTa papers attempted.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Umuvuduko n'igipimo. Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
  • Kugera no kugera. Yagura uburyo bwindimi nuburyo bwo gutumanaho.
  • Ibyemezo bisobanutse. Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.

Witondere

  • Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
  • Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
  • Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.

Intambwe ikurikira

  1. Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
  2. Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
  3. Komeza kugenzura abantu kugenzura ibisubizo byinshi.
  4. Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.

Inkomoko no gusoma

  1. arXiv (Yang et al.)XLNet: Generalized Autoregressive Pretraining for Language Understanding
  2. NeurIPS 2019 ProceedingsXLNet: Generalized Autoregressive Pretraining for Language Understanding
  3. arXiv / ACL 2019 (Dai et al.)Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context
  4. arXiv (Liu et al.)RoBERTa: A Robustly Optimized BERT Pretraining Approach
  5. GitHub (XLNet authors)zihangdai/xlnet

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the XLNet and Permutation Language Modeling quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is XLNet and Permutation Language Modeling?

XLNet is a 2019 language model from Carnegie Mellon University and Google Brain that learns bidirectional context without BERT's [MASK] token. It trains by predicting words in many randomly shuffled orders, a method called permutation language modeling, and it outperformed BERT on 20 benchmark tasks in its NeurIPS 2019 paper.

Niyihe ntego yo kwitegura XLNet ikoresha?

XLNet yerekana ibyateganijwe byinjira-byemewe hejuru yimpushya zose zerekana ibimenyetso, byitwa imvugo yemewe.

Ni izihe ntege nke za BERT XLNet yagenewe gukemura?

Ikimenyetso cya BERT [MASK] ntigishobora kugaragara mugihe cyo gutunganya neza kandi gifata ibyahanuwe bipfutse nkibigenga; XLNet yirinda ibibazo byombi.

Ni ubuhe butumwa bwa XLNet bwo-kwiyitaho butandukanye?

Ibirimo bikubiyemo ibimenyetso hamwe nibisobanuro, mugihe ikibazo cyibibazo kimenya intego yintego ariko ntigikubiyemo, gishobora guhanura nta kumeneka.

Ese XLNet ihindura umubiri mumagambo?

XLNet yemerera guhanura (factorisation) ikoresheje mask yo kwitondera; imyanya yumwimerere imyanya ibitswe binyuze muri kodegisi.

Ni ubuhe bwubatsi bukora nkumugongo wa XLNet kumurongo muremure?

XLNet yubaka kuri Transformer-XL, yongeraho igice gisubiramo hamwe na kodegisi ihagaze kugirango ikore ibintu birebire.