Daidaiton Tsare-tsaren Cikakkun Bayanai
Cikakken Sharded Data Parallel (FSDP) dabara ce ta horarwa da aka rarraba wacce ke raba sigogin samfuri, gradients, da jihohin ingantawa a cikin GPUs da yawa don haka kowace na'ura tana riƙe da yanki kawai.
Dubawa
It makes training huge models possible on hardware that could never fit the whole model in one GPU's memory.
Zurfafa nutsewa
Daidaiton bayanan al'ada yana kiyaye cikakken kwafin samfurin akan kowane GPU, wanda ke ɓarna ƙwaƙwalwar ajiya kuma yana ɗaukar girman ƙirar. FSDP, wanda __AIU_PROTECTED_13_'s PyTorch ya shahara kuma aka yi wahayi daga Microsoft's ZeRO, a maimakon haka ya ruguza abubuwa uku a cikin na'urori: sigogi, gradients, da jihohin ingantawa. A lokacin wucewar gaba, kowane GPU yana tattara cikakken ma'auni na ɗan lokaci don layin da yake lissafta ta hanyar taron duka, yana gudanar da lissafin, sannan nan da nan ya fitar da kwafin da aka tattara. Fassara ta baya tana aiki iri ɗaya, sannan mai rage-watsawa wanda ke rarraba yankan gradient zuwa ga GPUs nasu. Saboda kowace na'ura kawai tana adana ɗan juzu'in ƙirar ne ta dindindin, amfani da ƙwaƙwalwar ajiya yana faɗuwa daidai da adadin GPUs, yana barin ƙungiyoyi su horar da ƙira tare da dubun ko ɗaruruwan biliyoyin sigogi.
Fahimtar Fasaha
FSDP yana cinikin ƙarin sadarwa don ajiyar ƙwaƙwalwar ajiya. Ana sake gina ma'aunin kowane Layer bisa buƙatu tare da tattara gabaɗaya kai tsaye kafin amfani kuma a watsar da su nan da nan, yayin da ake haɗa gradients kuma a raba tare da rage-watse. Ana iya haɗa sadarwa tare da ƙididdigewa ta hanyar ƙaddamar da sigogin Layer na gaba yayin da Layer na yanzu ke gudana, yana ɓoye yawancin jinkirin hanyar sadarwa. Daidaita girman girman sharding (manufofin nannade) yana daidaita sawun ƙwaƙwalwar ajiya da kan sadarwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Cikakkun Bayanan Daidaitawa
FSDP yana zama tsoho don buɗe babban horo na samfuri, tare da FSDP2 a cikin PyTorch yana haɓaka amfani da kowane siga. Yi tsammanin haɗin kai mai ƙarfi tare da tensor da daidaiton bututun don samfuran siga- tiriliyan, mafi kyawun tallafi don gauraye daidaito da fp8, da mafi wayo na nannade atomatik wanda ke zaɓe muku iyakoki. Kamar yadda inter-GPU ke haɗuwa kamar NVLink da InfiniBand suna samun sauri, farashin sadarwa na sharding yana ci gaba da raguwa, yana mai da shi aiki a ma'auni mafi girma.
Aiwatar da Gaskiyar Duniya
Kyakkyawan daidaita samfurin Llama-biliyan 70 a cikin GPUs 8 waɗanda daban-daban ba zai iya ɗaukar cikakken ma'auni ba.
Horar da manyan samfuran harshe a ɗakunan gwaje-gwaje na AI ta hanyar rarraba jihohin ingantawa (waɗanda ke mamaye ƙwaƙwalwar ajiya tare da Adam) a cikin ɗaruruwan masu haɓakawa.
Masu bincike suna amfani da PyTorch's FSDP wrapper don horar da masu canji na hangen nesa akan rukunin jami'a ba tare da siyan flagship 80GB GPUs ba.
Haɗa FSDP tare da gauraye-daidai bfloat16 zuwa kusan rabin ƙwaƙwalwar ajiya da haɓaka aikin horarwa akan samfuran multimodal.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Fully Sharded Data Parallel quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Daidaiton Bayanai
Tambayoyin da ake yawan yi
What is Fully Sharded Data Parallel?
Cikakken Sharded Data Parallel (FSDP) dabara ce ta horarwa da aka rarraba wacce ke raba sigogin samfuri, gradients, da jihohin ingantawa a cikin GPUs da yawa don haka kowace na'ura tana riƙe da yanki kawai. Yana ba da damar horar da manyan samfura akan kayan masarufi waɗanda ba za su taɓa dacewa da duka ƙirar a cikin ƙwaƙwalwar GPU ɗaya ba.
Menene FSDP shard a fadin GPUs wanda daidaitaccen daidaiton bayanai baya yi?
FSDP ya ɓata ma'auni na ƙirar, gradients, da jihohin ingantawa a cikin na'urori, yayin da daidaitattun bayanai ke yin kwatankwacin cikakken samfurin akan kowane GPU.
Wane aiki gama gari FSDP ke amfani da shi don sake gina cikakken ma'aunin Layer kafin lissafta shi?
Kafin Layer ya gudana, FSDP na yin taro don tattara cikakkun sigogi na ɗan lokaci daga duk shards, sannan ya 'yantar da su daga baya.
Me yasa FSDP ke 'yantar da cikakkun ma'aunin nauyi nan da nan bayan lissafin Layer?
Rike shard kawai na dindindin da tattara cikakken ma'auni na wucin gadi shine abin da ke rage amfani da ƙwaƙwalwar ajiya ƙasa kuma yayi daidai da juzu'i ɗaya na samfurin.
FSDP an yi wahayi sosai ta wace hanya inganta ƙwaƙwalwar ajiya ta farko?
FSDP's Sharding na sigogi, gradients, da inganta jihohi suna bin ra'ayoyin da aka gabatar a cikin Microsoft's ZeRO daga DeepSpeed Library.
Ta yaya FSDP ke ɓoye yawancin jinkirin hanyar sadarwa daga tara nauyi?
FSDP tana ƙaddamar da sigogi na Layer na gaba yayin da Layer na yanzu ke ci gaba da ƙididdigewa, yana mamaye duk hanyar sadarwa tare da aiki mai amfani.