Imyitozo yihuta na Megatron
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka.
Incamake
Without them, today's frontier models simply could not fit in memory or finish training in a reasonable time.
Kwibira cyane
Guhugura icyitegererezo kinini kuri GPU imwe ntibishoboka kuko uburemere, gradients, hamwe na leta nziza ntabwo bihuye. Ibi bice byagabanije akazi muri GPU nyinshi. Megatron-LM yatangije uburinganire bwa tensor, ikata kugwiza matrix kugiti cye muri buri cyiciro hakurya ya GPU, hiyongereyeho imiyoboro ibangikanye, ishyira ibice bitandukanye kuri GPU zitandukanye. Umusanzu wa DeepSpeed ni ZeRO (Zero Redundancy Optimizer), igabanya optimizer leta, gradients, hamwe nibipimo muri GPU aho kuyigana, guca kuri buri GPU yibuka cyane. Byombi bikunze guhuzwa (Megatron-DeepSpeed) kugirango bitoze moderi nka BLOOM-176B na Megatron-Turing NLG. Bongeraho kandi kuvanga-neza, kugenzura ibikorwa, no gupakurura kuri CPU cyangwa NVMe kuburyo moderi nini zitoza kubikoresho bike.
Ubushishozi
ZeRO ifite ibyiciro bitatu byo kongera ubwizigame bwo kwibuka: Icyiciro cya 1 shards optimizer ivuga, Icyiciro cya 2 nacyo kigabanya gradients, naho icyiciro cya 3 kigabanya ibipimo ubwabyo, bikabikusanya kubisabwa mugihe cyimbere ninyuma. Ufatanije na tensor parallelism (intra-layer) hamwe no guhuza imiyoboro (inter-layer), iyi miterere '3D parallelism.' Impagarara zingenzi ni itumanaho hejuru: buri shard itandukanijwe yongeraho GPU-kuri-GPU, bityo injeniyeri zihuza ibice kugirango NVLink na InfiniBand byihute.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Kazoza Kumuvuduko Wihuse na Megatron Amahugurwa
Witegereze guhuza hamwe na FSDP kavukire ya PyTorch (Byuzuye Byuzuye Byuzuye) Abakoranya-bayobora hamwe nuburyo bwikora bwa parallelism bateganya gukuraho intoki. Mugihe ihuriro ryamahugurwa rigenda ryiyongera ryibihumbi magana yihuta, kwihanganira amakosa, gupima ibintu byoroshye, no gutumanaho gukabije hamwe no kubara biba imipaka yiganjemo ubwubatsi, hamwe no gushyigikira ibyuma bishya nka NVIDIA Blackwell hamwe na chip yo gutoza ibicuruzwa.
Gushyira mu bikorwa Isi
Kumenyereza gufungura indimi nyinshi BLOOM-176B ukoresheje uburyo bwa Megatron-DeepSpeed stack ihuriweho na magana ya GPU.
Microsoft na NVIDIA bahugura miliyari 530-ibipimo bya Megatron-Turing NLG hamwe na 3D ibangikanye.
ZeRO-Kureka kureka abashakashatsi bagahuza neza na miliyari-miriyari-yimiterere yimikorere kumurongo umwe GPU mukwirakwiza leta optimizer kuri RAM RAM.
Koresha igenzura rya enterineti muri ibi bice kugirango uhuze Windows ndende murwego rwo kubara ibikorwa aho kubika byose.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the DeepSpeed and Megatron Training Stacks quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
GPTQ na AWQ Nyuma yo Guhugura
Ibibazo bikunze kubazwa
What is DeepSpeed and Megatron Training Stacks?
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka. Bitabaye ibyo, imipaka yumupaka uyumunsi ntishobora guhuza kwibuka cyangwa kurangiza imyitozo mugihe gikwiye.
Niyihe ntego yibanze ya optimizer ya ZeRO ya DeepSpeed?
ZeRO (Zero Redundancy Optimizer) ikuraho ubudahangarwa bwibuke mugutandukanya leta, gradients, hamwe nibipimo muri GPU aho kuyigana kuri buri gikoresho.
Tensor parallelism, nkuko yabanjirijwe muri Megatron-LM, igabanya icyitegererezo gute?
Tensor parallelism igabanya imibare imbere murwego rumwe (nka matrix nini igwira) hejuru ya GPU nyinshi, ikaba igabanyijemo ibice.
Niki cyiciro cya ZeRO gitanga ububiko bukomeye bwo kuzigama no kugabanya ibipimo by'icyitegererezo ubwabo?
ZeRO Icyiciro cya 3 cyerekana ibipimo byiyongera kuri gradients na optimizer leta, kubiteranya kubisabwa, bigabanya kwibuka cyane.
Niki 'activation checkpointing' igurisha kure kugirango ubike kwibuka mugihe cy'amahugurwa?
Kugenzura ibikorwa byo kubika bibika ibikorwa bito bito kandi bigasubirwamo mugihe cyo gusubira inyuma, gucuruza iyindi mibare yo kugabanya kwibuka.
Kuki guhuza ubwo buhanga bakunze kwita 'parallelism ya 3D'?
3D parallelism ikubiyemo ingamba eshatu zifatika: guhuza amakuru, tensor (intra-layer) kubangikanya, hamwe numuyoboro (inter-layer).