Adam and Adaptive Optimizers
Adam shine mai inganta dokin aiki a bayan yawancin hanyoyin sadarwa na zamani, yana daidaita ƙimar koyo ta atomatik ga kowane siga.
Dubawa
It matters because it makes training deep models faster and far less finicky than plain gradient descent.
Zurfafa nutsewa
Adam (Adaptive Moment Estimation), wanda Kingma da Ba suka gabatar a cikin 2014, ya haɗa ra'ayoyi biyu. Na farko, kuzari: yana kiyaye matsakaicin ruɓewa na ɓangarorin abubuwan da suka wuce (lokacin farko) don haka sabuntawa suna haɓaka gudu cikin madaidaiciyar kwatance. Na biyu, sikelin kowane siga: yana bin matsakaicin matsakaicin murabba'in gradients (lokaci na biyu) kuma yana rarraba kowane mataki da tushen murabba'in waccan darajar, don haka sigogi tare da manyan gradients masu hayaniya suna ɗaukar ƙananan matakai kuma waɗanda ba a sabunta su ba suna ɗaukar matakai mafi girma. Wannan daidaitawa yana nufin sau da yawa zaka iya amfani da ƙimar koyo guda ɗaya a duk hanyar sadarwa. Bambance-bambancen, AdamW, yana kawar da lalata nauyi daga sabuntawar gradient kuma ya zama tsoho don horar da manyan tashoshi da ƙirar harshe.
Fahimtar Fasaha
Adamu yana kula da matsakaita guda biyu a kowace siga: m (gradients) da v (squared gradients), sabuntawa tare da ƙimar lalata beta1 (yawanci 0.9) da beta2 (yawanci 0.999). Domin duka biyun suna farawa daga sifili, ana gyara son zuciya ta hanyar rarraba ta (1 - beta ^ t). Sabuntawa shine theta = theta - lr * m_hat / (sqrt (v_hat) + epsilon), inda epsilon (a kusa da 1e-8) yana hana rarraba ta sifili. Wannan shine dalilin da ya sa Adamu yana buƙatar ƙaramin daidaita ƙimar koyo idan aka kwatanta da SGD a sarari.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Adamu da Masu ingantawa
Adam da AdamW sun kasance masu rinjaye, amma bincike yana tura inganci don ƙirar siga- tiriliyan, inda adana ƙarin ƙima biyu akan kowane nauyi yana da tsada. Bambance-bambancen hasken ƙwaƙwalwar ajiya kamar Adafactor, 8-bit Adam, da sabbin masu ingantawa kamar Lion (wanda ke amfani da ƙarfin tushen alamar kawai) da Sophia suna nufin daidaita ingancin Adamu tare da ƙarancin ƙwaƙwalwar ajiya ko haɗuwa cikin sauri. Yi tsammanin na'urori masu daidaitawa da aka kunna musamman don rarrabawa, horo mara ƙarancin inganci don ci gaba da haɓakawa.
Aiwatar da Gaskiyar Duniya
Horar da manyan nau'ikan harshe kamar GPT da Llama, waɗanda ke amfani da AdamW azaman madaidaicin ingantawa.
Kyakkyawan daidaita ma'aunin hoto da aka riga aka horar (misali, ResNet) akan saitin bayanai na al'ada tare da tsayayyen ƙimar koyan Adam kawai.
Horar da samfuran watsawa a bayan masu samar da hoto kamar Stable Diffusion.
Gudun 8-bit Adam a cikin ɗakunan karatu kamar bitsandbytes don dacewa da jihohin ingantawa cikin ƙayyadaddun ƙwaƙwalwar GPU.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Adam and Adaptive Optimizers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
ZeRO da Sharded Optimizers
Tambayoyin da ake yawan yi
What is Adam and Adaptive Optimizers?
Adam shine mai inganta dokin aiki a bayan yawancin hanyoyin sadarwa na zamani, yana daidaita ƙimar koyo ta atomatik ga kowane siga. Yana da mahimmanci saboda yana sa ƙirar ƙira mai zurfi da sauri kuma ba ta da ƙarfi fiye da zuriyar gradient.
Wadanne nau'i biyu ne Adamu ke bibiyar kowane siga?
Adamu yana kiyaye matsakaicin ruɓewa mai ƙarfi na gradients (lokacin farko) da na madaidaitan gradients (lokaci na biyu) ga kowane siga.
Me yasa Adamu ya yi amfani da gyara na son zuciya ga kiyasin lokacinsa?
Dukansu m da v an fara su ne zuwa sifili, don haka kiyasin farko ba su da son rai; raba ta (1 - beta^t) yana gyara wannan.
Menene babban bambanci tsakanin Adam da AdamW?
AdamW yana shafan rubewar nauyi kai tsaye zuwa ma'aunin nauyi maimakon haɗa shi cikin ma'auni mai daidaitawa, wanda ke haɓaka haɓakawa a cikin tasfofi.
Wace rawa ƙaramar kalmar epsilon ke takawa a cikin tsarin sabunta Adamu?
Ana ƙara Epsilon (a kusa da 1e-8) zuwa maƙasudin don kada sigogin da ke kusa da sifili-square-gradient ba su haifar da fashewa ba.
Me yasa ake yawan kwatanta Adamu a matsayin 'mai daidaitawa'?
Ta hanyar rarraba ta tushen murabba'in kowane madaidaicin murabba'in-girma, Adam yana daidaita girman matakin kowane siga maimakon amfani da ƙimar duniya ɗaya ɗaya.