Xiga xigaHagaha xiga
GPTQ iyo AWQ Tirinta Tababarka Kadib
Farsamo
HAGAHA Farsamada
DeepSpeed iyo Megatron Tababarka
DeepSpeed (Microsoft) iyo Megatron-LM (NVIDIA) waa xirmooyin software ka dhigaya moodooyinka tababbarka balaayiin cabbirro ah oo kumanaan GPUs ah oo run ahaantii macquul ah.
Dhageyso hagahanAI-waa laga sheekeeyay · ~2 min
Boggaan2 daqiiqo akhri
Dulmar
Without them, today's frontier models simply could not fit in memory or finish training in a reasonable time.
quusid qoto dheer
Tababarka nooc weyn oo hal GPU ah waa wax aan macquul aheyn sababtoo ah culeysyada, gradients, iyo gobolada wax hagaajiya kuma haboona. Xirmooyinkani waxay u kala qaybiyeen shaqada GPU-yo badan. Megatron-LM waxay hormood u noqotay isbarbardhigga tensor-ka, iyada oo jarjaraysa isku dhufashada shaxanka shakhsi ahaaneed gudaha lakab kasta GPU-yada oo dhan, iyo isbarbardhigga dhuumaha, kaas oo dhigaya lakabyo kala duwan GPU-yo kala duwan. Wax ku darsiga saxeexa DeepSpeed waa ZeRO (Zero Redundancy Optimizer), kaas oo jeexjeexaya dawladaha hagaajinta, gradients, iyo cabirrada guud ahaan GPU-yada halkii ay ku soo celin lahaayeen iyaga, si weyn u jaraya xusuusta GPU-ba. Labada badanaa waa la isku daraa (Megatron-DeepSpeed ) si loo tababaro moodooyinka sida BLOOM-176B iyo Megatron-Turing NLG. Waxay sidoo kale ku daraan saxnaanta isku dhafan, isbaarada kicinta, iyo u dajinta CPU ama NVMe si moodooyinka waaweyn ay u tababaraan qalab xaddidan.
Dhaqangelinta Adduunka-dhabta ah
Tababarka qaabka furan ee BLOOM-176B ee luuqadaha badan ku hadla iyadoo la adeegsanayo isku dhafka Megatron-DeepSpeed ee boqolaal GPU-yada ah.
Microsoft iyo NVIDIA waxay tababarayaan 530-bilyan-beere Megatron-Turing NLG model oo leh isbarbar 3D.
ZeRO-Offload waxay u oggolaanaysaa cilmi-baarayaashu inay hagaajiyaan moodooyinka balaayiin-beegyada-badan ee hal goob shaqo oo GPU ah iyagoo u daadinaya gobollada wax-qabad ee CPU RAM.
Isticmaalka isbaarada hawlgelinta ee xidhmooyinkan si aad ugu habboonaato daaqadaha macnaha guud adiga oo dib u xisaabinaya hawl-qabadyada halkii aad ku wada kaydin lahaydeen.
Mustaqbalka DeepSpeed iyo Tababarka Megatron
Filo is dhexgalka adag ee PyTorch's FSDP (Fully Sharded Data Parallel), kaas oo nuugay fikrado badan oo ZeRO ah, oo mugdi gelinaya xariiqda udhaxeysa xirmooyinka cilmi-baarista iyo qaab-dhismeedka asaasiga ah. Hababka isku-dubbaridka ah iyo qorshayaasha isbarbar-dhigga tooska ah waxay ujeeddadoodu tahay inay meesha ka saaraan hagaajinta gacanta. Marka kooxaha tababarku ay u koraan boqollaal kun oo dardar-geliyayaal ah, dulqaadka khaladka ah, cabbirka laastikada, iyo isgaarsiinta isdhaafsiga ee xisaabinta ayaa noqda xudduudaha injineernimada ee ugu sarreeya, oo ay weheliso taageerada qalab cusub sida NVIDIA Blackwell iyo chips tababbarka caadada ah.
Ku dhaqangelinta
Maxay muhiim u tahay
- Qiimaha iyo miisaaniyada. Go'aamada qaab-dhismeedku waxay horseedaan waxqabadka iyo kharashka hawlgalka sannadaha.
- Go'aamo cad. Waxbarashada farsamada waxay ka caawisaa kooxaha inay doortaan xidhmo sax ah, ma aha oo kaliya kan ugu cusub.
- Xakamaynta tayada. Doorashooyinka injineernimada ee wanaagsan waxay yareeyaan shilalka la isku halleyn karo ee wax soo saarka.
Iska ilaali
- Hagaajinta hal bartilmaameed waxay qarin kartaa daciifnimada nidaamka ballaaran.
- Kaabayaasha dhaqaalaha iyo dayactirka inta badan waa la dhayalsadaa.
- Nabadgelyada iyo daldaloolada u fiirsashada ayaa kori kara marka nidaamyadu noqdaan kuwo aad u adag.
Tallaabooyinka xiga
- Qeex daahida, tayada, iyo bartilmaameedyada qiimaha ka hor inta aan la hirgelin.
- Benchmark marka la eego culeyska dhabta ah iyo xaaladaha xogta.
- La socodka qalabka khaladaadka, leexashada, iyo saamaynta isticmaalaha.
- U diyaari dib-u-noqoshada iyo dariiqyada jawaab-celinta dhacdada ka hor inta aanad miisaan.
Sii wad Sahaminta
Warqad bilaash ah
Hel warbixinta maalinlaha ah ee AI
Saddex sheeko oo la xaqiijiyay oo AI ah maalin kasta oo shaqo ah, oo lagu qoray Ingiriis fudud. Bilaash had iyo jeer, aan xayaysiis lahayn.
Hal iimayl maalin kasta oo shaqo ah. Ka bax hal gujin. Marna ma iibinno ama ma wadaagno ciwaankaaga.
Is tijaabi
Take the DeepSpeed and Megatron Training Stacks quiz
Fikrad degdeg ah oo ku saabsan jawaab kasta, iyo shahaado la wadaajin karo oo leh aqoonsi la xaqiijin karo markaad ka gudubto koorso.
Taageer waxbarashada AI bilaashka ah. AI Understanding waa 501(c)(3) aan faa'iido doon ahayn — xayaysiis la'aan, albaab lacag ah ma jiro, had iyo jeer. Samee deeq
Su'aalaha soo noqnoqda
What is DeepSpeed and Megatron Training Stacks?
DeepSpeed (Microsoft) iyo Megatron-LM (NVIDIA) waa xirmooyin software ka dhigaya moodooyinka tababbarka balaayiin cabbirro ah oo kumanaan GPUs ah oo run ahaantii macquul ah. Iyaga la'aantood, moodooyinka xuduudaha maanta si fudud kuma qabsan karaan xusuusta mana dhamayn karaan tababarka waqti macquul ah.
Waa maxay ujeedada koowaad ee DeepSpeed's ZeRO optimizer?
ZeRO (Zero Redundancy Optimizer) waxay meesha ka saartaa dib-u-celinta xusuusta iyadoo la kala qaybinayo dawladaha wax hagaajiya, gradients, iyo cabbirrada guud ahaan GPU-yada halkii ay ku soo celin lahaayeen qalab kasta.
Isbarbardhigga Tensor-ka, sidii horudhac ugu ahaa Megatron-LM, u kala qaybiyaa qaabka sidee?
Isbarbardhigga Tensor-ku wuxuu u kala qaybiyaa xisaabta gudaha lakab keliya (sida matrix weyn oo ku dhufto) GPU-yo badan, kaas oo kala qaybsanaan-lakabka dhexdiisa.
Marxaladee ZeRO ayaa bixisa kaydinta xusuusta ugu weyn iyaga oo sidoo kale kala qaybinaya cabbirada tusaalaha laftooda?
ZeRO Stage 3 jeexjeexyada jaangooyooyinka marka lagu daro gradients iyo dawladaha hagaajinta, iyaga oo u soo ururinaya baahida, taasoo siinaysa dhimista xusuusta ugu weyn.
Maxay 'baaritaan hawlgelinta' ka beddeshaa si loo badbaadiyo xusuusta inta lagu jiro tababarka?
Isbaarada hawlgelinta waxay kaydisaa hawl-qabadyo dhexdhexaad ah oo yar waxayna dib u xisaabisaa inta lagu jiro faafinta, ka ganacsiga xisaabinta dheeraadka ah si loo yareeyo xusuusta.
Waa maxay sababta isku-darka farsamooyinkan inta badan loogu yeero 'isbarbardhigga 3D'?
Isbarbar dhigga 3D wuxuu xiraa saddex xeeladood oo toosan: isbarbardhigga xogta, isbarbardhigga tensor (lakabka gudaha) iyo dhuumaha (lakabka dhexda) barbar socda.
Sii wad waxbarashada
Tilmaamaha la xidhiidha
Tilmaamayaal badan ayaa loo doortay mawduucan