Incamake
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka. Bitabaye ibyo, imipaka yumupaka uyumunsi ntishobora guhuza kwibuka cyangwa kurangiza imyitozo mugihe gikwiye.
Amahugurwa ya DeepSpeed na Megatron nigikoresho cya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Guhugura icyitegererezo kinini kuri GPU imwe ntibishoboka kuko uburemere, gradients, hamwe na leta nziza ntabwo bihuye. Ibi bice byagabanije akazi muri GPU nyinshi. Megatron-LM yatangije uburinganire bwa tensor, ikata kugwiza matrix kugiti cye muri buri cyiciro hakurya ya GPU, hiyongereyeho imiyoboro ibangikanye, ishyira ibice bitandukanye kuri GPU zitandukanye. Umusanzu wa DeepSpeed ni ZeRO (Zero Redundancy Optimizer), igabanya optimizer leta, gradients, hamwe nibipimo muri GPU aho kuyigana, guca kuri buri GPU yibuka cyane. Byombi bikunze guhuzwa (Megatron-DeepSpeed) kugirango bitoze moderi nka BLOOM-176B na Megatron-Turing NLG. Bongeraho kandi kuvanga-neza, kugenzura ibikorwa, no gupakurura kuri CPU cyangwa NVMe kuburyo moderi nini zitoza kubikoresho bike.
Ubushishozi
ZeRO ifite ibyiciro bitatu byo kongera ubwizigame bwo kwibuka: Icyiciro cya 1 shards optimizer ivuga, Icyiciro cya 2 nacyo kigabanya gradients, naho icyiciro cya 3 kigabanya ibipimo ubwabyo, bikabikusanya kubisabwa mugihe cyimbere ninyuma. Ufatanije na tensor parallelism (intra-layer) hamwe no guhuza imiyoboro (inter-layer), iyi miterere '3D parallelism.' Impagarara zingenzi ni itumanaho hejuru: buri shard itandukanijwe yongeraho GPU-kuri-GPU, bityo injeniyeri zihuza ibice kugirango NVLink na InfiniBand byihute.
Kumenya Umuvuduko Wihuse na Megatron Amahugurwa
Umuvuduko wihuse (Microsoft) na Megatron-LM (NVIDIA) ni porogaramu ya software ikora icyitegererezo cyamahugurwa hamwe na miliyari y'ibipimo mubihumbi GPUs birashoboka. Bitabaye ibyo, imipaka yumupaka uyumunsi ntishobora guhuza kwibuka cyangwa kurangiza imyitozo mugihe gikwiye. Amahugurwa ya DeepSpeed na Megatron nigikoresho cya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata DeepSpeed na Megatron Training Stacks nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mubimenyerezo, amakipe akomeye akoresha DeepSpeed na Megatron Training Stack ahindura imyubakire, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Kumenyereza gufungura indimi nyinshi BLOOM-176B ukoresheje uburyo bwa Megatron-DeepSpeed stack ihuriweho na magana ya GPU.
Microsoft na NVIDIA bahugura miliyari 530-ibipimo bya Megatron-Turing NLG hamwe na 3D ibangikanye.
ZeRO-Kureka kureka abashakashatsi bagahuza neza na miliyari-miriyari-yimiterere yimikorere kumurongo umwe GPU mukwirakwiza leta optimizer kuri RAM RAM.
Koresha igenzura rya enterineti muri ibi bice kugirango uhuze Windows ndende murwego rwo kubara ibikorwa aho kubika byose.
Uburyo bwo Gushyira mu bikorwa
Imyitozo yimbitse na Megatron Imyitozo
Kumenyereza gufungura indimi nyinshi BLOOM-176B ukoresheje uburyo bwa Megatron-DeepSpeed stack ihuriweho na magana ya GPU.
Kumenyereza gufungura indimi nyinshi BLOOM-176B ukoresheje icyiciro cya Megatron-DeepSpeed ikomatanyirijwe hamwe mu magana ya GPUs Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbitse, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Imyitozo yimbitse na Megatron Imyitozo
Microsoft na NVIDIA bahugura miliyari 530-ibipimo bya Megatron-Turing NLG hamwe na 3D ibangikanye.
Microsoft na NVIDIA bahugura miliyari 530-yerekana ibipimo bya Megatron-Turing NLG hamwe na 3D parallelism Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbitse, kandi agakurikirana inyungu zibyara umusaruro nibiciro byigihe.
Imyitozo yimbitse na Megatron Imyitozo
ZeRO-Kureka kureka abashakashatsi bagahuza neza na miliyari-miriyari-yimiterere yimikorere kumurongo umwe GPU mukwirakwiza leta optimizer kuri RAM RAM.
ZeRO-Kureka kureka abashakashatsi bagahuza neza na miriyari-miriyari-yimiterere yimikorere kumurongo umwe wakazi GPU mugusuka leta optimizer mumakipe ya CPU RAM Ubusanzwe babona ibisubizo byiza mugihe basobanuye ibipimo byimbitse imbere, bagakomeza inzira yo kuzamura abantu kubibazo bikurikirana, kandi bagakurikirana inyungu zibyara umusaruro nibiciro byigihe.
Imyitozo yimbitse na Megatron Imyitozo
Koresha igenzura rya enterineti muri ibi bice kugirango uhuze Windows ndende murwego rwo kubara ibikorwa aho kubika byose.
Ukoresheje igenzura rya enterineti muri ibi bice kugirango uhuze Windows ndende murwego rwo kubara ibikorwa aho kubibika Amakipe yose mubisanzwe abona ibisubizo byiza mugihe asobanuye ibipimo byimbere imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbitse, kandi akurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.