Kasuwancin sake lissafin kunnawa
Ƙididdigar kunnawa (tambarin dubawa ko kunnawa) yana adana ƙwaƙwalwar GPU yayin horo ta hanyar watsar da kunnawa na tsaka-tsaki a cikin wucewar gaba da sake yin lissafin su yayin wucewar baya.
Dubawa
It trades extra compute for the ability to train larger models or longer sequences on the same hardware.
Zurfafa nutsewa
Bayar da baya yana buƙatar kunnawa-wuta don ƙididdige gradients, don haka ta tsohuwa ana adana kayan aikin kowane Layer - babban farashin ƙwaƙwalwar ajiya wanda ke girma tare da girman ƙirar, girman tsari, da tsayin jeri. Ƙididdigar kunnawa tana adana ƴan ƴan tantanin 'checkpoint' (sau da yawa kawai iyakoki) kuma yana watsar da sauran. A lokacin wucewar baya, yana sake gudanar da lissafin gaba tsakanin wuraren bincike don sabunta abubuwan da aka jefar akan buƙata. Sakamakon al'ada shi ne cewa tare da wuraren bincike da aka sanya kowane yadudduka na sqrt(N), ƙwaƙwalwar ajiya tana raguwa zuwa kusan O(sqrt (N)) yayin ƙara kusan ƙarin wucewa ɗaya na gaba (~ 33% ƙarin ƙididdigewa). Zaɓaɓɓun bambance-bambancen suna sake ƙididdige abubuwa masu arha-amma-ƙwaƙwalwa-nauyi (kamar hankali ko faduwa) yayin da ake adana masu tsada, samun mafi yawan ajiyar ƙwaƙwalwar ajiya don ƙarancin ƙididdigewa sama da ƙasa.
Fahimtar Fasaha
Babban ciniki shine ƙwaƙwalwar ajiya tare da FLOPs. Cikakken ƙididdigewa yana ƙara ƙarin wucewa gaba ɗaya kowane mataki (~ 30-40% a hankali) amma yana iya yanke ƙwaƙwalwar kunnawa ta tsari mai girma. Ɗauki mai wayo shine zaɓin dubawa: gano ops waɗanda ke da girman ƙwaƙwalwar ajiya amma ƙididdige-arha (softmax, Layernorm, GELU, ƙimar kulawa) kuma a sake lissafin waɗancan kawai, yayin da ake adana sakamakon GEMM masu tsada - rage girman ƙididdigewa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Canjin Sake lissafin Kunnawa
Sake lissafin yana ƙara sarrafa kansa da zaɓi. Tsarukan yanzu suna yin bayanin ƙwaƙwalwar ajiyar kowane op da farashin FLOP don zaɓar wuraren bincike mafi kyau, da haɗa ƙididdiga tare da ƙaddamar da kunnawa zuwa CPU/NVMe kuma tare da dabarun daidaitawa. Yayin da tsayin mahallin da girman samfurin ke ci gaba da girma, sa ran manufofin masu tarawa (a cikin PyTorch, JAX/XLA) waɗanda ke ɗaukar yanke shawara na kowane-op ta atomatik, tare da ƙarin juzu'i na sake ƙididdigewa tare da sadarwa don haka an ɓoye ƙarin FLOPs.
Aiwatar da Gaskiyar Duniya
Horar da babban taswira wanda ba zai dace ba ta hanyar duba kowane shingen Layer
Yin amfani da PyTorch's torch.utils.checkpoint don nannade tubalan wuta da yanke ƙwaƙwalwar kunnawa
Zaɓin sake lissafin hankali/softmax a cikin Megatron-LM don adana ƙwaƙwalwar ajiya tare da raguwa kaɗan
Bayar da tsayin jeri akan ƙayyadaddun kasafin kudin GPU ta hanyar sake lissafin kunnawa maimakon adana su.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Activation Recomputation Tradeoffs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
SmoothQuant da Ƙididdiga Kunnawa
Tambayoyin da ake yawan yi
What is Activation Recomputation Tradeoffs?
Ƙididdigar kunnawa (tambarin dubawa ko kunnawa) yana adana ƙwaƙwalwar GPU yayin horo ta hanyar watsar da kunnawa na tsaka-tsaki a cikin wucewar gaba da sake yin lissafin su yayin wucewar baya. Yana cinikin ƙarin ƙididdigewa don ikon horar da samfura masu girma ko tsayin jeri akan kayan masarufi iri ɗaya.
Menene cinikin sake kunna kunnawa don adana ƙwaƙwalwar ajiya?
Sake ƙididdigewa yana watsar da kunnawa da aka adana kuma yana sabunta su a cikin fasfo na baya, yana kashe ƙarin lissafi don rage amfanin ƙwaƙwalwar ajiya.
Me yasa a kullum ana adana abubuwan kunna-wuta kwata-kwata?
Fas ɗin baya yana amfani da kunnawa gaba don ƙididdige matakan gradients, don haka ta tsohuwa ana ajiye su a ƙwaƙwalwar ajiya har sai fas ɗin baya ya gudana.
Kusan nawa nawa ne ƙarin lissafin kunnawa ke ƙarawa?
Cikakken ƙididdigewa yana sake gudanar da lissafin gaba yayin wucewar baya, yana ƙara kusan ƙarin wuce gaba ɗaya - akan tsari na 30-40% ƙarin ƙididdigewa.
Menene ra'ayin da ke bayan zaɓe (ba cikakke) sake lissafin ba?
Zaɓuɓɓukan ƙididdige ƙididdige ƙididdiga na ops waɗanda ke amfani da ɗimbin ƙwaƙwalwar ajiya amma ƙananan ƙididdigewa (kamar softmax ko Layernorm), yayin da ake adana sakamakon GEMM masu tsada don rage ɓatattun FLOPs.
Wace dabarar da ake amfani da ita sau da yawa ana haɗawa da ƙididdiga don adana ƙarin ƙwaƙwalwar ajiya?
Ƙaddamar da kunnawa yana motsa wasu kunnawa zuwa ma'ajiyar CPU/NVMe, kuma ana haɗe shi akai-akai tare da ƙididdigewa da daidaitawa don ƙarin ajiyar ƙwaƙwalwar ajiya.