Ubuyobozi bwa tekiniki
Imyitozo yihuta na Megatron
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka.
Umva iki gitaboAI-byavuzwe · ~2 min
Kuriyi page2 min soma
Incamake
Bitabaye ibyo, imipaka yumupaka uyumunsi ntishobora guhuza kwibuka cyangwa kurangiza imyitozo mugihe gikwiye.
Kwibira cyane
Guhugura icyitegererezo kinini kuri GPU imwe ntibishoboka kuko uburemere, gradients, hamwe na leta nziza ntabwo bihuye. Ibi bice byagabanije akazi muri GPU nyinshi. Megatron-LM yatangije uburinganire bwa tensor, ikata kugwiza matrix kugiti cye muri buri cyiciro hakurya ya GPU, hiyongereyeho imiyoboro ibangikanye, ishyira ibice bitandukanye kuri GPU zitandukanye. Umusanzu wa DeepSpeed ni ZeRO (Zero Redundancy Optimizer), igabanya optimizer leta, gradients, hamwe nibipimo muri GPU aho kuyigana, guca kuri buri GPU yibuka cyane. Byombi bikunze guhuzwa (Megatron-DeepSpeed) kugirango bitoze moderi nka BLOOM-176B na Megatron-Turing NLG. Bongeraho kandi kuvanga-neza, kugenzura ibikorwa, no gupakurura kuri CPU cyangwa NVMe kuburyo moderi nini zitoza kubikoresho bike.
Gushyira mu bikorwa Isi
Kumenyereza gufungura indimi nyinshi BLOOM-176B ukoresheje uburyo bwa Megatron-DeepSpeed stack ihuriweho na magana ya GPU.
Microsoft na NVIDIA bahugura miliyari 530-ibipimo bya Megatron-Turing NLG hamwe na 3D ibangikanye.
ZeRO-Kureka kureka abashakashatsi bagahuza neza na miliyari-miriyari-yimiterere yimikorere kumurongo umwe GPU mukwirakwiza leta optimizer kuri RAM RAM.
Koresha igenzura rya enterineti muri ibi bice kugirango uhuze Windows ndende murwego rwo kubara ibikorwa aho kubika byose.
Kazoza Kumuvuduko Wihuse na Megatron Amahugurwa
Witegereze guhuza hamwe na FSDP kavukire ya PyTorch (Byuzuye Byuzuye Byuzuye) Abakoranya-bayobora hamwe nuburyo bwikora bwa parallelism bateganya gukuraho intoki. Mugihe ihuriro ryamahugurwa rigenda ryiyongera ryibihumbi magana yihuta, kwihanganira amakosa, gupima ibintu byoroshye, no gutumanaho gukabije hamwe no kubara biba imipaka yiganjemo ubwubatsi, hamwe no gushyigikira ibyuma bishya nka NVIDIA Blackwell hamwe na chip yo gutoza ibicuruzwa.
Kubishyira mubikorwa
Impamvu ari ngombwa
- Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
- Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
- Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Witondere
- Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
- Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
- Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Intambwe ikurikira
- Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
- Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
- Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
- Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Ibaruwa y'amakuru y'ubuntu
Ibuka amakuru ya buri munsi ya AI
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Take the DeepSpeed and Megatron Training Stacks quiz
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Ibibazo bikunze kubazwa
Ni ubuhe buryo bwimbitse bwihuta na Megatron?
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka. Bitabaye ibyo, imipaka yumupaka uyumunsi ntishobora guhuza kwibuka cyangwa kurangiza imyitozo mugihe gikwiye.
Niyihe ntego yibanze ya optimizer ya ZeRO ya DeepSpeed?
ZeRO (Zero Redundancy Optimizer) ikuraho ubudahangarwa bwibuke mugutandukanya leta, gradients, hamwe nibipimo muri GPU aho kuyigana kuri buri gikoresho.
Tensor parallelism, nkuko yabanjirijwe muri Megatron-LM, igabanya icyitegererezo gute?
Tensor parallelism igabanya imibare imbere murwego rumwe (nka matrix nini igwira) hejuru ya GPU nyinshi, ikaba igabanyijemo ibice.
Niki cyiciro cya ZeRO gitanga ububiko bukomeye bwo kuzigama no kugabanya ibipimo by'icyitegererezo ubwabo?
ZeRO Icyiciro cya 3 cyerekana ibipimo byiyongera kuri gradients na optimizer leta, kubiteranya kubisabwa, bigabanya kwibuka cyane.
Niki 'activation checkpointing' igurisha kure kugirango ubike kwibuka mugihe cy'amahugurwa?
Kugenzura ibikorwa byo kubika bibika ibikorwa bito bito kandi bigasubirwamo mugihe cyo gusubira inyuma, gucuruza iyindi mibare yo kugabanya kwibuka.
Kuki guhuza ubwo buhanga bakunze kwita 'parallelism ya 3D'?
3D parallelism ikubiyemo ingamba eshatu zifatika: guhuza amakuru, tensor (intra-layer) kubangikanya, hamwe numuyoboro (inter-layer).
Komeza wige
Kuyobora
Abandi bayobozi batoranijwe kuriyi ngingo