Fexe Rollout ak Dagg Bopp
Attention rollout anam la wuñuy toppee ni xibaar bi di jaar ci biir Transformer bi ngir leeral ban jeton dugal mooy indi jafe-jafe ci ab wax luy waaja am.
Résumé
Head pruning removes attention heads that contribute little, shrinking models without hurting accuracy. Together they help us interpret and compress Transformers.
Plongeur bu xóot
Transformers yi dañu tasaare seen xalaat ci boppu attention yu bari ci layer yu bari, kon benn layer buy bàyyi xel ci kàrtu bariwul lumuy nettali jaar-jaar bi yépp. Attention rollout, bi Abnar ak Zuidema dugal ci 2020, dafay saafara lii ci yokk matrices attention layer par layer (ginnaaw ñu xayma lëkkaloo yi des) ngir xayma ba ñaata token bu nekk ci duggal bi mujjee am ci benn token bu genn. Ci beneen wall bi, ay gestu yu mel ni Michel ak ay naataango 'Ndax fukk ak jiroom benn bopp ñoo gëna baax benn?' wane nañu ni bopp yu bari dañuy redondance: fraction bu mag mën nañu ko dagg ci diiru inference ak perte bu tuuti ci njub. Dagg bopp dafay rang bopp yi ci solo, di faral di jëfandikoo poñ yu lalu ci sensitivite bu lalu ci gradient, ba noppi maske yi gëna néew njariñ. Ñaari pexe yooyu dañuy àndaale: rollout dafay wane ban wàll ci reso bi moo gëna am solo ngir tekki, ak dagg jëf ci redondance ngir defar model yu gëna ndaw ak gëna gaaw.
Gis-gis xarala
Attention rollout dafay jàppee bàyyi xel ci bépp couche ni matrix transition, yokk ci benn composant dàntite ngir modele lëkkaloo skip residuel bi, normalise liiñ yi, ba noppi yokk matrice yooyu ci couche yi ngir am influens cumulatif token-to-token. Dagg bopp dafay xayma solo bu bopp bu nekk, lu bari ci gradient bi ñuy seentu ci perte bi ci wàllu masku bopp bu mën soppiku, ba noppi di zero bopp yi am poñ yu néew. Ñoom ñaar ñépp a ngi aju ci jumtukaayu modular buy bàyyi xel ci bopp yu bari.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Rollout ak Dagg Bopp
Lu model yi di gëna màgg, inference yu baax ak leeral yu wóor ñoom ñaar dañu gëna am solo. Xaarandil boppu dagg ngir boole ak dagg structurel, kantite, ak distillation ci tuyo deployment ngir am boor ak njëg yu sensible. Tekki-tekki mingi awaase ginaaw rollout ci wàllu bàyyi xel, pexe gradient-pondéré, ak jàngat sircuit mekanik buy saytu fonction boppu bu nekk. Fitnaal bi ñuy yamale ngir juntuwaay yu bees yi dina wey di dawal ay gestu yuy lëkkale njiit yi am solo ak li ñuy xayma ci dëgg.
Doxal ci àdduna dëgg
Xalaatal ban baat ci benn frase la Transformer classifier sukkandikoo ci, ci joxe xel ngir wane token yu am doole
Komprime ab xeetu BERT ngir jëfandikoo mobile ci dagg boppu yiy bàyyi xel ci lu bari ngir wàññi latency
Saytu ab xeetu njuumte ci topp limu bàyyi xel ci ab wax luy waaja am dellu ci ay token dugal yu am solo
Gaawaale gis-gis ci sistemu tekki làkk yi ci dindi bopp yu amul solo yiñ xàmmee ci poñ yiñ joxe ci sensitivite
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Attention Rollout and Head Pruning quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Bopp yu bari yu nëbbu
Laaj yi ñuy faral di laaj
What is Attention Rollout and Head Pruning?
Attention rollout anam la wuñuy toppee ni xibaar bi di jaar ci biir Transformer bi ngir leeral ban jeton dugal mooy indi jafe-jafe ci ab wax luy waaja am. Dagg bopp di dindi xel bopp yi def lu néew, xeetu shrinking te du gaañ njub. Ñu bokk jàppale ñu tekki ak kompresse Transformers.
Luy mébetu fësal bàyyi xel ci nit ñi?
Rollout dafay yokk bàyyi xel ci diisaay yi (ak residu yi) ngir xayma ni token bu nekk ci duggal bi di amee njeexital ci genn gi.
Lan moo waral benn kàrtu bàyyi xel ci benn couche du doy ngir joxe leeral?
Ndax xalaat yi dañu séddaloo ci ay couche ak bopp yuñ dajale, benn kàrtu couche mënul jàpp lépp luy jaar ci xibaar bi.
Lan la rollout attention di yokk ci matrix attention bu nekk ngir leeral lëkkaloo yi des?
Yokk benn lëkkale dàntite dafay wane lëkkaloo skip residual biy may token yi ñu tëye seeni leeral.
Lan la gëstub bàyyi xel ci bopp yu bari wane ci redundancy bopp?
Jàngat yu melni 'Ndax fukk ak juróomi bopp ñoo gëna baax benn bopp?' wane nañu ni paccu bopp yu bari dañuy redondance ci inference.
naka lañuy xaymaa njariñu bopp ci dagg garab?
Njariñ dañu koy faral di joxe poñ ci jëfandikoo gradient bi ci ñàkk ci wàllu mask bu mën soppiku ci bopp bu nekk.