DeepSpeed na Megatron Ọzụzụ Stacks
DeepSpeed (Microsoft) na Megatron-LM (NVIDIA) bụ sọftụwia sọftụwia na-eme ka ụdị ọzụzụ nwere ijeri paramita gafere puku kwuru puku GPU ga-ekwe omume.
Nchịkọta
Without them, today's frontier models simply could not fit in memory or finish training in a reasonable time.
Ime miri emi
Ịzụ nnukwu ihe nlereanya na otu GPU agaghị ekwe omume n'ihi na ihe ọ̀tụ̀tụ̀ dị arọ, gradients, na steeti ndị na-ebuli elu adabaghị. Nchịkọta ndị a kewara ọrụ n'ofe ọtụtụ GPU. Megatron-LM sụrụ ụzọ tensor parallelism, na-egbutu ọnụọgụ matriks nke ọ bụla n'ime oyi akwa ọ bụla n'ofe GPUs, yana myirịta pipeline, nke na-etinye ọkwa dị iche iche na GPU dị iche iche. Ntinye mbinye aka DeepSpeed bụ ZeRO (Zero Redundancy Optimizer), nke shards optimizer steeti, gradients, na parampat gafee GPU kama ịmegharị ha, belata ebe nchekwa GPU ọ bụla nke ọma. A na-ejikọta ha abụọ (Megatron-DeepSpeed ) iji zụọ ụdị dịka BLOOM-176B na Megatron-Turing NLG. Ha na-agbakwunyekwa nkenke-agwakọta-nkenke, nyocha ọrụ, na mbupụ na CPU ma ọ bụ NVMe nnukwu ụdị na-azụ na ngwaike nwere oke.
Nghọta nka nka
ZeRO nwere usoro atọ nke ịbawanye nchekwa nchekwa ebe nchekwa: Stage 1 shards optimizer states, Stage 2 also shards gradients, na Stage 3 shards the paramates n'onwe ha, na-achịkọta ha na-achọrọ n'oge ngafe na azụ. Ejikọtara ya na tensor parallelism (intra-layer) na pipeline parallelism (inter-layer), nke a na-etolite '3D parallelism.' Isi esemokwu bụ nzikọrịta ozi n'elu: nkewa shard ọ bụla na-agbakwunye okporo ụzọ GPU-na-GPU, yabụ ndị injinia na-atụgharị nkewa ahụ iji mee ka njikọ NVLink na InfiniBand zuo ngwa ngwa.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke DeepSpeed na Megatron Ọzụzụ Stacks
Na-atụ anya njikọta siri ike na FSDP nke PyTorch (Fully Sharded Data Parallel), nke na-etinye uche n'ọtụtụ echiche ZeRO, na-eme ka ahịrị dị n'etiti nchịkọta nyocha na usoro isi. Ụzọ ndị nchịkọta na-achịkọta na ndị na-eme atụmatụ myirịta akpaaka na-achọ iwepụ nlegharị anya ntuziaka. Ka ụyọkọ ọzụzụ na-eto ruo ọtụtụ narị puku ndị na-eme ngwa ngwa, nnabata mmejọ, ngbanwe na-agbanwe agbanwe, na nkwurịta okwu jikọtara ọnụ na mgbakọ na-aghọ akụkụ kachasị nke injinia, yana nkwado maka ngwaike ọhụrụ dị ka NVIDIA Blackwell na ibe ọzụzụ omenala.
Mmejuputa n'ezie n'ụwa
Ọzụzụ ụdị BLOOM-176B nwere ọtụtụ asụsụ mepere emepe site na iji nchịkọta Megatron-DeepSpeed jikọtara ọnụ gafere ọtụtụ narị GPU.
Microsoft na NVIDIA na-azụ ụdị Megatron-Turing NLG nke ijeri ijeri 530 nwere myirịta 3D.
ZeRO-Offload na-ahapụ ndị nyocha ka ha dezie ụdị ọnụọgụ ijeri ijeri na otu GPU na-arụ ọrụ site n'ịkwasa steeti ndị na-eme ka ọ dịkwuo mma na CPU RAM.
Iji nbanye nleba anya n'ime ngwugwu ndị a iji dabaa ogologo okirikiri windo site n'ịgbakọ ọrụ kama ịchekwa ha niile.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the DeepSpeed and Megatron Training Stacks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
GPTQ na AWQ Ngụkọta Ọzụzụ gasịchara
Ajụjụ a na-ajụkarị
What is DeepSpeed and Megatron Training Stacks?
DeepSpeed (Microsoft) na Megatron-LM (NVIDIA) bụ sọftụwia sọftụwia na-eme ka ụdị ọzụzụ nwere ijeri paramita gafere puku kwuru puku GPU ga-ekwe omume. Na-enweghị ha, ụdị ókèala nke oge a enweghị ike dabara na ebe nchekwa ma ọ bụ mechie ọzụzụ n'oge kwesịrị ekwesị.
Kedu ebumnuche bụ isi nke DeepSpeed's ZeRO optimizer?
ZeRO (Zero Redundancy Optimizer) na-ewepụ mgbapụta ebe nchekwa site na ikewa steeti optimizer, gradients, na paramita n'ofe GPU kama ịmegharị ha na ngwaọrụ ọ bụla.
Tensor parallelism, dị ka ọsụ ụzọ na Megatron-LM, si kewaa ihe nlereanya?
Myirịta Tensor na-ekewa mgbakọ na mwepụ n'ime otu oyi akwa (dị ka nnukwu matriks na-amụba) n'ofe GPU dị iche iche, nke na-ekewa intra-layer.
Kedu ọkwa ZeRO na-enye nchekwa nchekwa kachasị ukwuu site na ịkekọrịta paramita ihe atụ n'onwe ha?
ZeRO Stage 3 shards paradita na mgbakwunye na gradients na steeti ndị na-eme ka ọ dịkwuo mma, na-achịkọta ha na ọchịchọ, na-enye mbelata ebe nchekwa kachasị.
Kedu ihe na-atụgharị 'activation checkpointing' iji chekwaa ebe nchekwa n'oge ọzụzụ?
Ntụle nlele mgbanaka na-echekwa ịgbalite n'etiti ole na ole ma na-atụgharị ha n'oge mgbasa ozi, na-azụ ahịa mgbako agbakwunyere maka ebe nchekwa belata.
Kedu ihe kpatara eji ejikọta usoro ndị a na-akpọkarị '3D parallelism'?
Myirịta 3D na-achịkọta atụmatụ atọ n'akụkụ atọ: myirịta data, tensor (intra-layer) myirịta, na pipeline (inter-layer) myirịta.