I-Warmup ne-Cosine Annealing Shedules
I-Warmup ngobumnene inyusa izinga lokufunda lisuka eduze kukaziro ngaphambi kokuqeqeshwa, bese i-cosine annealing iyibolisa kahle ihlehle ilandele ijika le-cosine.
Uhlolojikelele
Together they stabilize early training and squeeze out better final accuracy, which is why nearly every modern transformer is trained this way.
I-Deep Dive
Lapho ukuqeqeshwa kuqala, izisindo eziyimodeli azihleliwe futhi ama-gradient angaba mkhulu, ngakho-ke ukugxumela uqonde esilinganisweni esikhulu sokufunda ngokuvamile kubangela ukukhuphuka kokulahlekelwa noma ukwehlukana - ikakhulukazi ngezilungiseleli eziguquguqukayo ezifana no-Adam, izilinganiso zokuhluka kwazo ezingathembeki ezinyathelweni zokuqala. I-Warmup ilungisa lokhu ngokukhuphula izinga lesilinganiso esingaphezu kwezinyathelo ezingamakhulu ambalwa ukuya ezinkulungwaneni ezimbalwa. Uma imodeli isimi onyaweni oluzinzile, i-cosine annealing ithatha izintambo, ibola izinga elingu-0.5 * (1 + cos(pi * t / T)) lokuphakama kwayo. Umumo we-cosine ugcina isilinganiso siphezulu kusenesikhathi ukuze kuthuthuke ngokushesha, bese siyakhululeka kancane kancane ukuze isikhuthazi sikwazi ukuzinza sibe ngobuncane obuhle esikhundleni sokugxuma sizungeze.
I-Technical Insight
I-Cosine annealing ikala izinga lokufunda ngo-0.5 * (1 + cos(pi * t / T)), lapho u-t eyisinyathelo samanje futhi u-T eyingqikithi. Lokhu kuchitha isikhathi eside eduze nenani eliphakeme, kubola ngokushesha phakathi, bese kucaba eduze kukaziro ekugcineni - ngokungafani nokubola komugqa oqondile. I-Warmup ivamise ukuba yimigqa futhi imfushane. Ijika elihlanganisiwe libukeka njengegquma elibushelelezi: phezulu, i-plateau-ish, bese kuba ukushibilika okuthambile kuya cishe kuqanda.
I-Strategic Impact
Izindleko kanye nesabelomali
Izinqumo zezakhiwo ziqhuba ukusebenza kanye nezindleko zokusebenza iminyaka.
Izinqumo ezicacile
Imfundo yobuchwepheshe isiza amaqembu ukuthi akhethe isitaki esifanele, hhayi nje esisha.
Ukulawulwa kwekhwalithi
Izinketho ezingcono zobunjiniyela zinciphisa izehlakalo ezinokwethenjelwa ekukhiqizeni.
Ikusasa Le-Warmup ne-Cosine Annealing Shejuli
I-Warmup-plus-cosine ihlala iyiresiphi ezenzakalelayo yamamodeli olimi amakhulu, kodwa okuhlukile kuyasabalala. I-Warmup-stable-decay (WSD) igcina isilinganiso esingaguquki bese ibola ngokucijile ekugcineni, ikwenze kube lula ukunweba ukugijima ngaphandle kokuzibophezela kubude obumisiwe. Abacwaningi futhi bayatadisha ukuthi kungani i-warmup isebenza - ukuyixhuma nomsindo we-gradient kanye ne-loss-landscape curvature - futhi amathuluzi aya ngokuya eshuna ngokuzenzakalelayo ubude be-warmup nezinga eliphakeme, ehlisa isilingo nephutha lemanuwali elibusa namuhla.
Ukuqaliswa Komhlaba Wangempela
Amamodeli olimi esitayela se-GPT kanye nesitayela se-BERT asebenzisa i-warmup yomugqa ngaphezu kwezinyathelo zokuqala ezingu-~1-2% ezilandelwa ukubola kwe-cosine ukuya kuziro.
I-Vision transformers (ViT) isitimela esine-cosine annealing kanye ne-warmup emfushane ukugwema ukuhlukana kwangaphambi kwesikhathi ku-ImageNet.
I-Hugging Face Transformers inikeza `get_cosine_schedule_with_warmup` njengesihleli somugqa owodwa wemisebenzi yokushuna kahle.
I-Stable Diffusion namanye amamodeli okusabalalisa alungiswa kahle nge-warmup ukuze kuvinjelwe ukuqhuma kwe-gradient lapho kulungiswa izisindo eziqeqeshwe kusengaphambili.
Izingozi & Guardrails
Ukuthuthukisa ibhentshimakhi eyodwa kungafihla ubuthakathaka obubanzi besistimu.
Izindleko zengqalasizinda nezokulungisa zivame ukubukelwa phansi.
Izikhala zokuphepha nokubonakala zingakhula njengoba izinhlelo ziba nzima kakhulu.
Ukuqalisa Umhlahlandlela
Chaza ukubambezeleka, ikhwalithi, nezindleko ezihlosiwe ngaphambi kokuqaliswa.
Ibhentshimakhi ngaphansi komthwalo wangempela nezimo zedatha.
Ukuqapha amathuluzi amaphutha, ukukhukhuleka, nomthelela wabasebenzisi.
Lungiselela izindlela zokuhlehlisa nezigameko ngaphambi kokukala.
Qhubeka Uhlole
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Warmup and Cosine Annealing Schedules quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Umhlahlandlela olandelayo
Ukunciphisa Ukuqaphela Ubukhali
Imibuzo evame ukubuzwa
What is Warmup and Cosine Annealing Schedules?
I-Warmup ngobumnene inyusa izinga lokufunda lisuka eduze kukaziro ngaphambi kokuqeqeshwa, bese i-cosine annealing iyibolisa kahle ihlehle ilandele ijika le-cosine. Ndawonye bazinzisa ukuqeqeshwa kwangaphambi kwesikhathi futhi bakhiphe ukunemba okungcono kokugcina, yingakho cishe yonke i-transformer yesimanje iqeqeshwa ngale ndlela.
Iyini inhloso eyinhloko yesigaba se-warmup ekuqaleni kokuqeqeshwa?
Ukuqala ngenani elikhulu lokufunda ezisindweni ezingahleliwe kungabangela ukukhuphuka kokulahlekelwa noma ukwehlukana, ngakho-ke i-warmup inyusa izinga ngobumnene ukuze kugcinwe izinyathelo zangaphambi kwesikhathi zizinzile.
I-Cosine annealing ibola izinga lokufunda ngokulandela yimuphi umumo?
Izinga likalwa ngo-0.5 * (1 + cos(pi * t / T)), likhiqiza igquma elibushelelezi elihlala liphakeme kusenesikhathi futhi lishibilika lize lifinyelele kuziro ekugcineni.
Isiphi isilungiseleli esizuza kakhulu ku-warmup ngoba izilinganiso zaso zokuhluka azithembeki kusenesikhathi?
Izilinganiso zika-Adam zokwehluka okusebenzayo zisekelwe kumasampuli ambalwa kakhulu ezinyathelweni zokuqala, okwenza usayizi wesinyathelo osebenzayo ube nephutha - i-warmup inciphisa lokhu.
Kufomula ye-cosine, u-T umeleni?
T iwumkhathizwe lapho izinga libola lisuka phezulu lize lifinyelele kuqanda; t isinyathelo samanje kulowo mkhathizwe.
Iyiphi inzuzo eyodwa yokuhluka kwe-warmup-stable-decay (WSD) kune-cosine yobude obugxilile?
I-WSD ibamba isilinganiso esingaguquki bese ibola kakhulu ekugcineni, ukuze ukwazi ukugcina isigaba esizinzile sihamba isikhathi eside futhi unqume indawo yokuma kamuva.