XLNet Modeling
XLNet dafay boole muy tekki ñaari yoon ci BERT ak waxtaanu autoregressive bu GPT ci tàggat ci njubluwaayi baat yu bari.
Résumé
This permutation trick lets it learn from all positions without ever masking tokens.
Plongeur bu xóot
XLNet, Carnegie Mellon ak Brain ñoo ko dugal ci 2019, dañu ko tëral ngir saafara jafe-jafe bi ci tàggat yaram bu njëkk ci xeetu BERT. BERT dafay maske jeton yi te di leen wax luy am, waaye màndarga [MASK] bi ñu defar du musa feeñ ci waxtu fine-tuning, sos train / test mismatch, te BERT dafay jàpp ni jeton maske yi moom seen bopp. XLNet lu moy loolu dafay jëfandikoo 'modeling làkku permutation': dafay yokk log-likelihood bi ñuy seentu ci bépp njuréef bu mëna am ci baat yi ci benn toppalante. Suñu xasee ba jeton bu nekk ñu jox ko benn subset bu bari ci ñeneen ñi, model bi dafay gis bu baax contexte bidirectional ci noonu mu des ci model autorégressif bu baax te amul benn maske. Tabax ci kaw Transformer-XL ngir memory bu sori, XLNet moo raw BERT ci luy tollu ci 20 liggéey yu ci melni tontu laaj, jàngat yëg-yëg, ak rang dokimaa.
Gis-gis xarala
XLNet du jaxase kàddu yi; Dafay soppi doxalinu faktorisasioŋ bi jaaraleko ci masku bàyyi xel, suko defee leerali position yi dañuy denc. Ngir mëna def lii, dafay jëfandikoo 'ñaari stream self-attention': stream ëmbiit buy kode token bi ak contexte bi, ak stream laaj bu xam barabu target bi waaye xamul ëmbiit li, loolu mooy tax ñu mëna wax luy waaja am te du joxe tontu li. Baamtu Transformer-XL ak enkodaasu positional relatif daf koy may memory ci segment yu gudd, gëna yombal jëfandikoo dokimaa yu gudd.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu modelu weccoo XLNet
XLNet firnde la woon bu am solo ni mébetu autoregressif yi mën nañu jàpp muy tekki ci ñaari yoon, di dindi xaajaloo bi am ci diggante BERT ak GPT. Bi barab bi gëna bari luñu ko boole ci encodeur yu maske wala decodeur yu mag yuy régressive, xalaatu permutation XLNet ak Transformer-XL recurrence leeraloon nañu liggéey ci ginaaw ci modeling contexte bu yàgg ak mébetu pretraining buñ boole. Li muy gis mingi wéy di am solo ndax gëstukat yi dañuy wut architecture yu boole modeling contexte bu dëgër ak generation bu baax te amul mask.
Doxal ci àdduna dëgg
Def ay resilta yu gëna baax ci laaj-tontu benchmark yu melni SQuAD
Liggéeyu këyit yu gudd yu melni test jàng-xam RACE jaaraleko ci mémoire Transformer-XL
Sistemu rang dokimaa yu am doole ak seetlu xibaar
Yokkatal xeetu yëg-yëg ak xeetu mbind ci kaw BERT
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the XLNet Permutation Modeling quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Modeling topic
Laaj yi ñuy faral di laaj
What is XLNet Permutation Modeling?
XLNet dafay boole muy tekki ñaari yoon ci BERT ak waxtaanu autoregressive bu GPT ci tàggat ci njubluwaayi baat yu bari. Bii pexe permutation daf koy may mu jàng ci bépp position te du musa maske ay jeton.
Lan mooy mébetu tàggat-yaram bi XLNet di jëfandikoo?
XLNet dafay yokk li ñuy seentu ci log-likelihood ci bépp weccoo bu jëm ci doxalinu faktorisasioŋ token, ñu koy woowe modelu làkku weccoo.
Ban ñakk kattan ci BERT lañu jagleel XLNet ngir saafara?
[MASK] màndarga artifisiyel bu BERT musul feeñ ci diiru fine-tuning te dafay jàppee maskeer yi ñuy wax luy waaja am ni lu moom seen bopp; XLNet moytu na ñaari jafe-jafe yi.
Lan mooy tàqale XLNet ci ñaari xeeti bàyyi xel ci sa bopp?
Ëmbiit li dafay kode token bi ak contexte bi, waaye stream laaj bi xamna barabu target bi waaye xamul ëmbiit li, loolu mooy tax ñu mëna wax luy waaja am te du am leakage.
Ndax XLNet dafay jaxase kàddu yi ci santaane?
XLNet dafay soppi doxalinu wax luy waaja am (factorisasioŋ) jaaraleko ci masku bàyyi xel; position token yi njëkk ñu ngi leen di denc jaaraleko ci encodage positionnel.
Ban architecture mooy nekk yaxu ndigg ci XLNet ci wàllu sori?
XLNet dafa tabax ci kaw Transformer-XL, biy yokk segment yu bari ak enkodaasu position relatif ngir mëna jëfandikoo ay toppalante yu gudd.