Ubusanzwe
Ubusanzwe urwego rushyiraho amahugurwa mugukuraho ibikorwa muri buri rugero rwihariye kuburyo bafite zeru zeru nibice bitandukanye.
Incamake
It is a quiet but essential ingredient that makes deep transformers trainable.
Kwibira cyane
Yatangijwe na Ba, Kiros, na Hinton mu mwaka wa 2016, ubusanzwe (LayerNorm) bukemura ikibazo ko ibikorwa biri mumurongo wimbitse bishobora gutemba ku munzani itandukanye kuko ibimenyetso binyura mubice byinshi, bigatinda cyangwa bigahagarika imyigire. Bitandukanye nicyiciro gisanzwe, gisanzwe buri kintu murugero muri mini-batch, LayerNorm isanzwe kurwego rwibintu bimwe. Ibi bituma itigenga kubunini bwicyiciro kandi ikoreshwa kimwe mumahugurwa no gufata umwanzuro, kandi ikora muburyo busanzwe hamwe nuburebure bwikurikiranya, niyo mpamvu yabaye ihame rya transformateur ikoresha imiterere yindimi zigezweho. Nyuma yo gukora ibisanzwe, ikoresha igipimo cyiga (gamma) na shift (beta) kugirango urusobe rushobore kugarura ibyo rukeneye byose.
Ubushishozi
Kubiranga vector x, LayerNorm ibara ibisobanuro nibitandukanya hejuru yibyo bikoresho, hanyuma isohoka gamma * (x - bivuze) / sqrt (variance + epsilon) + beta. Kuberako imibare ituruka kumurongo umwe, imyitwarire irasa niba itsinda rifite ingero 1 cyangwa 1000. Impinduka yoroshye, RMSNorm, gusimbuka bisobanura gukuramo no kugabana gusa numuzi-bivuze-kare, kubika kubara; ikoreshwa muri moderi nka Llama. Gushyira kandi bifite akamaro: 'pre-norm' (isanzwe mbere ya buri sublayer) ituma impinduka zimbitse zoroha cyane imyitozo kuruta 'post-norm'.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza h'ibisanzwe
Ubusanzwe burimo kunozwa kugirango bikore neza murwego. RMSNorm yasimbuye ahanini LayerNorm muburyo bushya bw'ururimi runini kuko bihendutse kandi ikora kimwe, kandi gushyira-pre-norm isanzwe ubu nibisanzwe kubintu byimbitse. Abashakashatsi bakomeje gushakisha inyubako zidafite ubusanzwe zikoresha uburyo bwitondewe bwo gutangiza cyangwa gupima amayeri aho, bagamije guca hejuru mugihe bagumya guhugura imyitozo isanzwe itanga.
Gushyira mu bikorwa Isi
Gutezimbere buri gihinduramatwara muburyo bwururimi nka GPT na BERT.
Gushoboza RMSNorm nkuburyo bworoshye bwo guhitamo imbere muri moderi ya Llama-umuryango.
Guhindura ibipimo-birebire bikurikirana amakuru mumvugo hamwe nubusemuzi aho ingano yicyiciro itandukanye.
Emerera imyitozo yizewe hamwe nubunini bwicyiciro kimwe, nko mubice bimwe byo gushimangira imyigire.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Layer Normalization quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
RMSNorm na Pre-Layeri bisanzwe
Ibibazo bikunze kubazwa
What is Layer Normalization?
Ubusanzwe urwego rushyiraho amahugurwa mugukuraho ibikorwa muri buri rugero rwihariye kuburyo bafite zeru zeru nibice bitandukanye. Nibintu bituje ariko byingenzi bituma impinduka zimbitse zitozwa.
Hirya no hino kurwego rusanzwe rugereranya iki kandi rutandukanye?
LayeriNorm isanzwe hejuru yimiterere yikigereranyo cyurugero rumwe, bigatuma itigenga nizindi ngero mugice.
Ni ukubera iki ibisanzwe bisanzwe bikundwa kuruta icyiciro gisanzwe muri transformateur?
Kuberako imibare yacyo ituruka kurugero rumwe, LayerNorm yitwara neza ititaye ku bunini bwicyiciro kandi ikwiranye nimpinduka-ndende yinyandiko ikurikiranye.
Niki ibipimo byiga gamma na beta reka LayerNorm ikora?
Nyuma yo gukora ibisanzwe kuri zeru bisobanura no gutandukanya ibice, umunzani wa gamma na beta bihindura ibisubizo kugirango moderi idahatirwa kugabanwa kugenwe.
Nigute RMSNorm itandukaniye na Layeri isanzwe?
RMSNorm ikuraho intambwe igereranya intambwe nubunzani ukoresheje imizi-bivuze-kare-ya-ibikorwa, bihendutse kandi bikoreshwa muri moderi nka Llama.
Ni ikihe kibazo gisanzwe gisanzwe gifasha gukemura mumiyoboro yimbitse?
Nkuko ibimenyetso binyura mubice byinshi igipimo cyabo gishobora guturika cyangwa kugabanuka; ibisanzwe bikomeza ibikorwa murwego ruhamye kugirango gradients yitware neza.