DeepSpeed da Megatron Training Stacks
DeepSpeed (Microsoft) da Megatron-LM (NVIDIA) su ne tarin software waɗanda ke yin ƙirar horarwa tare da biliyoyin sigogi a cikin dubban GPUs a zahiri mai yiwuwa.
Dubawa
Without them, today's frontier models simply could not fit in memory or finish training in a reasonable time.
Zurfafa nutsewa
Horar da babban samfuri akan GPU ɗaya ba zai yuwu ba saboda ma'aunin nauyi, gradients, da jahohin ingantawa ba su dace ba. Waɗannan tarin sun raba aikin zuwa yawancin GPUs. Megatron-LM ya fara aikin tensor parallelism, yana yanka nau'ikan nau'ikan matrix a cikin kowane Layer a cikin GPUs, da daidaiton bututun mai, wanda ke sanya yadudduka daban-daban akan GPUs daban-daban. Gudunmawar sa hannu ta DeepSpeed ita ce ZeRO (Zero Redundancy Optimizer), wanda ke shards masu haɓaka jihohi, gradients, da sigogi a cikin GPUs maimakon yin kwafin su, yanke ƙwaƙwalwar kowane-GPU sosai. Sau da yawa ana haɗa su biyun (Megatron-DeepSpeed ) don horar da samfura kamar BLOOM-176B da Megatron-Turing NLG. Hakanan suna ƙara daidaito-daidaitacce, wurin duba kunnawa, da saukarwa zuwa CPU ko NVMe don haka manyan samfuran suna horar da ƙayyadaddun kayan aiki.
Fahimtar Fasaha
ZeRO yana da matakai uku na haɓaka ajiyar ƙwaƙwalwar ajiya: Stage 1 shards optimizer states, Stage 2 kuma shards gradients, da Stage 3 shards da sigogi da kansu, tattara su a kan bukatar lokacin wucewa gaba da baya. Haɗe da tensor parallelism (intra-Layer) da kuma daidaitaccen bututu (inter-Layer), wannan yana samar da 'daidaicin 3D.' Makullin tashin hankali shine sadarwa sama da sama: kowane tsagewar shard yana ƙara zirga-zirgar GPU-zuwa-GPU, don haka injiniyoyi suna daidaita rarrabuwar don kiyaye hanyoyin NVLink da InfiniBand cikin sauri.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar DeepSpeed da Horarwar Horarwar Megatron
Yi tsammanin haɗin kai tare da FSDP na asali na PyTorch (Fully Sharded Data Parallel), wanda ya mamaye ra'ayoyin ZeRO da yawa, yana ɓatar da layi tsakanin tarin bincike da ainihin tsarin. Hanyoyi masu haɗawa da masu tsara layi na atomatik suna nufin cire kunnawa na hannu. Yayin da gungu na horarwa ke girma zuwa ga ɗaruruwan dubunnan masu haɓakawa, haƙurin kuskure, ƙima mai ƙarfi, da sadarwa tare da ƙididdigewa sun zama manyan iyakokin injiniya, tare da tallafi don sabbin kayan masarufi kamar NVIDIA Blackwell da guntun horo na al'ada.
Aiwatar da Gaskiyar Duniya
Horar da samfurin BLOOM-176B na buɗe harshe da yawa ta amfani da haɗin Megatron-DeepSpeed a cikin ɗaruruwan GPUs.
Microsoft da NVIDIA suna horar da siga-Biliyan 530 na Megatron-Turing NLG tare da daidaiton 3D.
ZeRO-Offload yana barin masu bincike su daidaita samfuran siga-biliyoyin-biyu akan GPU guda ɗaya ta hanyar zubar da jihohin ingantawa zuwa CPU RAM.
Yin amfani da wurin duba kunnawa a cikin waɗannan ɗigon don dacewa da tsawon windows mahallin ta hanyar sake lissafin kunnawa maimakon adana su duka.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the DeepSpeed and Megatron Training Stacks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
GPTQ da AWQ Ƙididdigar Koyarwa Bayan Koyarwa
Tambayoyin da ake yawan yi
What is DeepSpeed and Megatron Training Stacks?
DeepSpeed (Microsoft) da Megatron-LM (NVIDIA) su ne tarin software waɗanda ke yin ƙirar horarwa tare da biliyoyin sigogi a cikin dubban GPUs a zahiri mai yiwuwa. Idan ba tare da su ba, ƙirar iyakoki na yau ba za su iya dacewa da ƙwaƙwalwar ajiya ba ko gama horo a cikin madaidaicin lokaci.
Menene ainihin manufar DeepSpeed's ZeRO ingantawa?
ZeRO (Zero Redundancy Optimizer) yana kawar da raguwar ƙwaƙwalwar ajiya ta hanyar rarraba jihohin ingantawa, gradients, da sigogi a cikin GPUs maimakon maimaita su akan kowace na'ura.
Daidaitawar Tensor, kamar yadda aka yi majagaba a Megatron-LM, ya raba samfurin ta yaya?
Daidaitawar Tensor yana raba lissafin lissafi a cikin Layer guda (kamar babban matrix yana ninka) a cikin GPUs da yawa, wanda shine rarrabuwar ciki-Layer.
Wane mataki na ZeRO yana ba da mafi girman ajiyar ƙwaƙwalwar ajiya ta hanyar raba sigogin ƙirar da kansu?
ZeRO Stage 3 sigogi shards ban da gradients da jihohin ingantawa, tara su akan buƙata, yana ba da mafi girman raguwar ƙwaƙwalwar ajiya.
Menene cinikin 'kunnawa' don adana ƙwaƙwalwar ajiya yayin horo?
Ma'anar kunna kunnawa tana adana ƙarancin kunnawa na tsaka-tsaki kuma tana ƙididdige su yayin yaɗa baya, cinikin ƙarin ƙididdiga don rage ƙwaƙwalwar ajiya.
Me yasa ake yawan haɗa waɗannan fasahohin da ake kira 'parallelism 3D'?
Daidaitawar 3D ta tattara dabaru iri-iri uku: daidaiton bayanai, daidaitawar tensor (intra-Layer), da daidaiton bututun (inter-Layer).