Jadawalin Warmup da Cosine Annealing
Warmup a hankali yana haɓaka ƙimar koyo daga kusa da sifili kafin horo, sa'an nan kuma cirewar cosine a hankali yana lalatar da shi baya bin lankwasa na cosine.
Dubawa
Together they stabilize early training and squeeze out better final accuracy, which is why nearly every modern transformer is trained this way.
Zurfafa nutsewa
Lokacin da aka fara horarwa, ma'aunin ƙira yana da bazuwar kuma gradients na iya zama babba, don haka tsalle kai tsaye zuwa babban ƙimar koyo yakan haifar da asara ko rarrabuwa - musamman tare da inganta haɓakawa kamar Adam, waɗanda ƙididdige bambance-bambancen su ba abin dogaro bane a matakan farko. Warmup yana gyara wannan ta hanyar haɓaka ƙimar sama da ƴan ɗari zuwa ƴan matakai dubu. Da zarar samfurin ya kasance akan ƙaƙƙarfan ƙafar ƙafa, annealing cosine yana ɗaukar nauyi, yana lalata ƙimar kamar 0.5 * (1 + cos (pi * t / T)) na kololuwar sa. Siffar cosine tana kiyaye ƙimar girma da wuri don ci gaba cikin sauri, sannan a sauƙaƙe kashewa a hankali don inganta haɓakawa zai iya daidaitawa cikin mafi ƙarancin ƙarami maimakon yin bouncing a kusa da shi.
Fahimtar Fasaha
Cosine annealing yana daidaita ƙimar koyo da 0.5 * (1 + cos(pi * t / T)), inda t shine mataki na yanzu kuma T shine duka. Wannan yana ɗaukar lokaci mai tsawo a kusa da ƙimar kololuwa, yana raguwa da sauri a tsakiya, sannan ya faɗi kusa da sifili a ƙarshen - ba kamar ruɓar layi madaidaiciya ba. Warmup yawanci layi ne kuma gajere. Haɗe-haɗe yana kama da tudu mai santsi: sama, plateau-ish, sa'an nan mai laushi mai laushi zuwa kusan sifili.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Warmup da Jadawalin Annealing na Cosine
Warmup-plus-cosine ya kasance babban girke-girke na manyan harsuna, amma bambance-bambancen suna yaduwa. Warmup-stable-decay (WSD) yana kiyaye ƙimar akai-akai sannan ya lalace sosai a ƙarshe, yana sauƙaƙa tsawaita gudu ba tare da sake komawa zuwa tsayayyen tsayi ba. Masu bincike kuma suna nazarin dalilin da yasa dumama ke aiki - haɗa shi zuwa ƙarar ƙararrawa da karkatar da yanayin ƙasa - da kuma kayan aikin da ke ƙara daidaita tsayin dumama ta atomatik da ƙimar kololuwa, rage gwajin gwajin-da-kuskuren da ke mamaye yau.
Aiwatar da Gaskiyar Duniya
Salon GPT da nau'in yare na BERT suna amfani da dumama mai layi akan farkon ~ 1-2% na matakan da lalata cosine ke bi zuwa kusa da sifili.
Masu canjin hangen nesa (ViT) suna horar da cosine annealing da ɗan gajeren dumi don guje wa bambance-bambancen farko akan ImageNet.
Hugging Face Transformers yana ba da `samun_cosine_schedule_with_warmup` azaman mai tsara layi ɗaya don daidaita ayyukan.
Stable Diffusion da sauran nau'ikan nau'ikan yaduwa suna daidaita kyau tare da dumama don hana fashe fashe lokacin daidaita ma'aunin da aka riga aka horar.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Warmup and Cosine Annealing Schedules quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Ragewar Kaifi-Arewa
Tambayoyin da ake yawan yi
What is Warmup and Cosine Annealing Schedules?
Warmup a hankali yana haɓaka ƙimar koyo daga kusa da sifili kafin horo, sa'an nan kuma cirewar cosine a hankali yana lalatar da shi baya bin lankwasa na cosine. Tare suna daidaita horon da wuri kuma suna fitar da ingantaccen daidaito na ƙarshe, wanda shine dalilin da ya sa kusan kowane taswirar zamani ana horar da su ta wannan hanyar.
Menene babban manufar lokacin dumi a farkon horo?
Farawa da babban ƙimar koyo akan ma'aunin bazuwar na iya haifar da asara ko rarrabuwa, don haka dumama yana haɓaka ƙimar a hankali don kiyaye matakan farko.
Cosine annealing yana lalata ƙimar koyo bayan wane siffa?
Ana ƙididdige ƙimar da 0.5 * (1 + cos(pi * t / T)), yana samar da tudu mai santsi wanda ke tsayawa da wuri kuma yana tafiya kusa da sifili a ƙarshe.
Wanne mai ingantawa musamman yana amfana daga dumama saboda bambance-bambancensa ba su da dogaro da wuri?
Ƙididdiga na bambance-bambancen da ɗan Adam ke gudana ya dogara ne akan wasu samfura kaɗan a cikin matakan farko, suna yin ingantaccen girman matakin da ba daidai ba - dumama yana rage wannan.
A cikin tsarin cosine, menene T yake wakilta?
T shine sararin sama wanda adadin ke raguwa daga samansa zuwa kusa da sifili; t shine mataki na yanzu a cikin wannan sararin sama.
Menene fa'ida ɗaya na bambance-bambancen-stable-decay (WSD) sama da ƙayyadaddun ƙayyadaddun ƙayyadaddun ƙayyadaddun ƙaya?
WSD yana riƙe da ƙima akai-akai sannan ya lalace sosai a ƙarshe, don haka zaku iya ci gaba da tsawan lokaci kuma ku yanke shawarar tsayawa daga baya.