Duban hankali na Gradient
Ƙididdiga ta ƙasa (wanda kuma ake kira maƙasudin kunnawa) dabara ce ta adana ƙwaƙwalwar ajiya wacce ke watsar da mafi yawan kunnawa na tsaka-tsaki yayin wucewar gaba da sake ƙididdige su akan tashi yayin yaɗa baya.
Dubawa
It lets you train deeper, larger networks by trading extra compute for much lower memory use.
Zurfafa nutsewa
Horar da hanyoyin sadarwa na jijiyoyi galibi suna adana ayyukan kunnawa kowane Layer yayin wucewar gaba saboda yada baya yana buƙatar su don ƙididdige gradients. Don ƙira mai zurfi waɗannan kunnawa suna mamaye ƙwaƙwalwar ajiya. Ma'aunin bincike a hankali a maimakon haka yana adana kunnawa kawai a cikin ɗimbin saiti na yadudduka 'Checkpoint' kuma yana watsar da sauran. Lokacin da backprop ya isa yankin da aka daina kunnawa, yana sake aiwatar da lissafin gaba don kawai wannan ɓangaren don sake haɓaka abin da yake buƙata, sannan ya ci gaba. Tare da wuraren bincike da aka sanya kusan kowane murabba'i-tushen-na-N, ƙwaƙwalwar ajiya don kunnawa tana raguwa daga oda N don yin odar murabba'in-na-N, yayin da ƙididdigewa ya tashi da kusan ƙarin wucewar gaba ɗaya kawai (kusan 20-30% a hankali). Wannan yana ba da damar dacewa da manyan nau'ikan batch ko mafi zurfi masu canzawa akan GPU iri ɗaya.
Fahimtar Fasaha
Dabarar tana amfani da cinikin lokaci-da-memori. Ajiye duk kunnawa yana da sauri amma ƙwaƙwalwar ƙwaƙwalwa-yunwa; sake lissafta su yana da arha akan na'urori na zamani dangane da tsadar ƙarancin ƙwaƙwalwar ajiya. Tsari kamar PyTorch (torch.utils.checkpoint) nannade wani module don haka ana iya yin lissafin abubuwan da ke cikin sa a baya. Zaɓin abubuwan wurin bincike: ko da tazara na kusan sassan sqrt(N) yana rage girman ƙwaƙwalwar ajiya yayin ƙara ƙarin faci ɗaya kawai na ƙididdige gabaɗaya.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Matsayin Bincike na Gradient
Binciken gradient yanzu daidai yake a cikin babban horon samfuri kuma yana ƙara sarrafa kansa, tare da ɗakunan karatu waɗanda ke zaɓar mafi kyawun wuraren bincike a gare ku. Yana haɗa nau'i-nau'i ta halitta tare da FSDP, gauraye daidaitattun, da saukewa don tura girman samfurin sama. Yi tsammanin binciken 'zaɓaɓɓen' wanda ke ƙididdige ayyuka masu arha kawai yayin adana masu tsada (kamar matrices mai hankali) cache, da hanyoyin da aka sarrafa mai tarawa a cikin kayan aikin kamar PyTorch's torch.compile waɗanda ke yanke shawarar abin da za a adana ta atomatik tare da ƙididdigewa don mafi kyawun ma'aunin ƙwaƙwalwar sauri.
Aiwatar da Gaskiyar Duniya
Horar da na'ura mai zurfi tare da girman tsari mai girma akan GPU guda ta hanyar watsar da sake lissafin kunnawa Layer.
Kyakkyawan samfurin hangen nesa akan hotuna masu tsayi inda taswirorin kunnawa zasu mamaye ƙwaƙwalwar GPU.
Rungumar Face Transformers yana ba da damar gradient_checkpointing=Gaskiya ya dace da ƙirar sigar biliyan-biliyan yayin daidaitawa.
Haɗa wuraren bincike tare da FSDP don haka duka sigogi da kunnawa ana kiyaye su ƙanana, suna ba da damar horar da samfuran manyan harshe.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Checkpointing quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Rarraba Gradient
Tambayoyin da ake yawan yi
What is Gradient Checkpointing?
Ƙididdiga ta ƙasa (wanda kuma ake kira maƙasudin kunnawa) dabara ce ta adana ƙwaƙwalwar ajiya wacce ke watsar da mafi yawan kunnawa na tsaka-tsaki yayin wucewar gaba da sake ƙididdige su akan tashi yayin yaɗa baya. Yana ba ku damar horar da zurfafa, manyan cibiyoyin sadarwa ta hanyar yin ciniki da ƙarin ƙididdigewa don ƙarancin amfani da ƙwaƙwalwar ajiya.
Menene ma'aunin binciken gradient da farko yana kasuwanci don adana ƙwaƙwalwar ajiya?
Ƙididdigar matakan bincike tana sake ƙididdige abubuwan kunnawa da aka jefar yayin faɗuwar baya, tare da kashe ƙarin ƙididdigewa don musanya don ƙarancin ƙwaƙwalwar ajiya.
Me yasa a kullum ake adana abubuwan kunnawa yayin wucewar gaba?
Backprop yana lissafta gradients ta amfani da matsakaicin kunnawa daga fasfo na gaba, don haka dole ne su kasance a cikin su sai dai idan an sake ƙididdige su.
Kusan ta yaya ma'aunin ƙwaƙwalwar kunna kunnawa idan an sanya wuraren bincike kowane yadudduka sqrt(N) a cikin hanyar sadarwar N-Layer?
Matsakaicin tazara game da kowane murabba'in tushen-na-N yadudduka yana rage žwažwalwar ajiyar kunnawa da aka adana daga oda N zuwa oda sqrt(N).
Kusan nawa nawa ne ƙarin ƙididdigewa ke ƙarawa madaidaicin madaidaicin madaidaicin ƙarawa?
Tare da kyakkyawan wurin duba wuri, saman yana kusan ƙarin wucewar gaba guda ɗaya, galibi kusan raguwar 20-30%.
A cikin PyTorch, wanne kayan aiki ne aka saba amfani da shi don amfani da matakan bincike na gradient zuwa module?
torch.utils.checkpoint yana kunshe da module don haka ana ƙididdige abubuwan kunnawa na ciki yayin baya maimakon a adana su.