ELECTRA Tàggat bu njëkk
ELECTRA anam la wu gëna am njariñ ngir tàggat modeli làkk yi ci jàngal leen ñu gis kàddu yu baaxul yi, duñu xalaat kàddu yu nëbbu yi.
Résumé
It matches BERT's quality using a fraction of the compute.
Plongeur bu xóot
ELECTRA (Jàngale bu baax benn encoder buy xaaj token yu ñuy wecci ci anam wu jaar yoon), bi Google ak Stanford dugal ci 2020, dafa wecci liggéeyu modelu làkk bu BERT ak 'gis token buñ wecci.' Reseau generateur bu ndaw dafay weccoo yenn baat yi ci benn frase ngir am yeneen baat yuñ mëna nangu, ba noppi model bu mag bi (discriminateur bi) dafay jàng jël dogal, ci token bu nekk, ndax original la wala ñu wecci ko. Ndax model bi dafay tàggat ci token yépp, du ~ 15% bi BERT maske, dafay jàng lu gëna gaaw. ELECTRA-Small dafa gëna am doole ci GPT bi ñu tàggat ak 30x gëna bari ordinatër, ba noppi ELECTRA-Large moo gëna am doole RoBERTa ak XLNet ci benchmark GLUE bi ñuy jëfandikoo lu tollu ci ñeent ci ordinatër yi.
Gis-gis xarala
Ñaari transformatër ñoo bokk tàggat. Generatër bi dafay def modelu làkk buñ maskeer ba noppi digle ay jeton yuñ wara wecci; diskriminatër bi dafay def xaaj bu ñaar (real vs. wecci) ci bépp posisioŋ. Li gëna am solo mooy, ñàkk gi ñu ngi koy xayma ci jeton yépp, du ci ñi maske rek, di joxe siñaal jàng bu gëna dëgër. Ñaari embeddings token yi bokk, generator bi dafay nekk lu ndaw (dafay faral di nekk lu tollu ci ñeent ci genn-wàll ci dayo discriminateur bi), ginaaw bi ñu ko tàggatee ñu sànni generator bi - discriminateur bi kese lañuy gëna defar ci suufu dex gi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu tàggat yaram ELECTRA
Xalaatu ELECTRA biy wecci-token-detection indi na ay coppite ci encodeur yu am doole yu melni DeBERTa-v3, ñu boole ko ak bàyyi xel bu amul benn werante ngir am resultaa yi gëna xarañ. Ginaaw mbootaay yi dañu gëna bàyyi xel ci njëgu tàggat yaram ak emprent karbon, mébetu tàggat yaram buy tàqale siñaal ci bepp token dafay wéy di xëcc nit ñi ngir tabax encoder yu dëgër te kompact. Xaarandil ni approche bi di wéy di yëgal model yu ndaw, yu gaaw yi ngir seetlu ci aparey bi, xaaj, ak seet fu model yu mag yi di gëna bari.
Doxal ci àdduna dëgg
Dafay gaaw ci xaaj mbind ak jàngat yëg-yëg fu ñu soxla encoder bu dëgër te jaar yoon
Di nekk yax ci wàllu seet ak sistemu rang dokimaa
Ajustement bu baax ELECTRA-Small ngir liggéey ci aparey wala NLP yu néew latency ak ordinatër yu néew
Liggéey ni encoder bu dëgër ngir xàmmee entite yu am tur ak tontu laaj-tontu benchmarks yu melni SQuAD ak GLUE
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the ELECTRA Pretraining quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
XLNet Modeling
Laaj yi ñuy faral di laaj
What is ELECTRA Pretraining?
ELECTRA anam la wu gëna am njariñ ngir tàggat modeli làkk yi ci jàngal leen ñu gis kàddu yu baaxul yi, duñu xalaat kàddu yu nëbbu yi. Dafay méngoo ak kalite BERT ci jëfandikoo benn wàll ci xayma bi.
Ban liggéey la ELECTRA di jëfandikoo ci palaasu modelu làkk bu maske?
ELECTRA dafay tàggat model bi ngir xam ban jeton lañu wecci ak generatër, du wax luy am ci kàddu yu maske.
Lu tax ELECTRA gëna am njariñ ci xayma BERT?
Discriminateur bi dafay xaaj token bu nekk ni réel wala ñu wecci ko, kon model bi dafay jàng ci 100% ci position yi ci barabu subset biñ maske rek.
Ban cër la reso generatër bu ndaw bi di def ci ELECTRA?
Generatër bi dafay def modelu làkk buñ maskeer ba noppi di joxe ay token yu baaxul, di sos liggéey bi diskriminatër bi wara def.
Lan mooy dall generatër bi ginaaw bi ñu tàggatee ba noppi?
Discriminator bi kese lañuy tëye ak defar ko bu baax ngir liggéey yi ci suuf; generatër bi dañu ko sànni.
Gëstukat yu bawoo ci ban mbootaay ñoo njëkka defar ELECTRA?
ELECTRA was introduced in 2020 by researchers at Google and Stanford University.