Incamake
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho. Nibyo bituma imyitozo yindimi nini yindimi hamwe na miliyari amagana yibipimo bishoboka.
Icyitegererezo na Pipeline Iringaniza ni tekinoroji yubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo.
Kwibira cyane
Model parallelism igabanya icyitegererezo kimwe kuri GPU nyinshi kuburyo ntamuntu numwe ukeneye gufata uburemere bwose. Hariho uburyohe bubiri. Tensor (intra-layer) parallelism igabanya imibare imbere murwego, nko gutema matrix nini yo kugwiza muri GPU buri gice kibara igice gisohoka. Umuyoboro (inter-layer) parallelism igenera ibice bitandukanye bikurikiranye kuri GPU zitandukanye, bityo igipande cya 1 kibaho kuri GPU 0, guhagarika 2 kuri GPU 1, nibindi, hamwe nibikorwa byanyuze imbere nkumurongo winteko. Ikibazo hamwe na pipine ya naive ni 'bubble': mugihe GPU 0 ikora mugice cya mbere, epfo na ruguru GPU yicaye ubusa. Pipelining igabanya buri cyiciro mo micro-batch kugirango ibyiciro byose bigume bihuze, bitezimbere imikoreshereze.
Ubushishozi
Tensor parallelism (nko muri NVIDIA Megatron-LM) igabanya matrices yuburemere inkingi- cyangwa umurongo-ikoresha kandi ikoresha byose-kugabanya kugirango ihuze ibisubizo byigice, ikomeza itumanaho imbere muri NVLink yihuta. Imiyoboro ibangikanye (GPipe, PipeDream) igabanya icyiciro muri micro-batch zinyura mubyiciro muri gahunda itangaje, bigabanya igihe 'bubble' kidafite akamaro. Byombi bikunze gutondekwa hamwe, hamwe na tensor parallelism muri node hamwe numuyoboro uhuza ibice.
Kumenya Icyitegererezo hamwe nu muyoboro
Iyo icyitegererezo ari kinini cyane kuburyo kidahuye na GPU imwe, icyitegererezo hamwe numuyoboro ugereranije bigabanya icyitegererezo ubwacyo mubikoresho. Nibyo bituma imyitozo yindimi nini yindimi hamwe na miliyari amagana yibipimo bishoboka. Icyitegererezo na Pipeline Iringaniza ni tekinoroji yubaka tekinike igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa mubipimo. Kugirango wubake byimbitse, fata Model hamwe na Pipeline Parallelism nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mu myitozo, amakipe akomeye akoresha Model na Pipeline Parallelism itezimbere ubwubatsi, amakuru, hamwe nibikorwa remezo birwanya kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPU binyuze muri tensor parallelism.
Gukoresha GPipe kugirango ushire ibice bitandukanye byerekezo nini cyangwa icyerekezo cyururimi kuri moteri yihuta mugihe micro-batching ituma bahuze.
Imashini ya moteri ya DeepSpeed igabanya moderi ya miliyari-ijana-yerekana ibipimo mubice byinshi.
Gukomatanya tensor parallelism imbere ya seriveri imwe ya 8-GPU hamwe na parallelisme ihuriweho na seriveri nyinshi kugirango itoze icyitegererezo kinini cyane kumashini imwe.
Uburyo bwo Gushyira mu bikorwa
Icyitegererezo na Pipeline Kuringaniza mubikorwa
Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPU binyuze muri tensor parallelism.
Guhugura imiterere-yuburyo bwa GPT hamwe na NVIDIA Megatron-LM, igabanya buri cyerekezo cya transformateur hamwe na matrica yo kugaburira imbere muri GPUs binyuze muri tensor parallelism Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Icyitegererezo na Pipeline Kuringaniza mubikorwa
Gukoresha GPipe kugirango ushire ibice bitandukanye byerekezo nini cyangwa icyerekezo cyururimi kuri moteri yihuta mugihe micro-batching ituma bahuze.
Gukoresha GPipe kugirango ushireho ibice bitandukanye byicyerekezo kinini cyangwa icyitegererezo cyururimi kuri moteri yihuta mugihe mugihe micro-bathing ituma bakora cyane Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Icyitegererezo na Pipeline Kuringaniza mubikorwa
Imashini ya moteri ya DeepSpeed igabanya moderi ya miliyari-ijana-yerekana ibipimo mubice byinshi.
Moteri ya DeepSpeed igabanya moderi ya miliyari-magana-yerekana urugero mu byiciro byinshi Amakipe akunze kubona umusaruro mwiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Icyitegererezo na Pipeline Kuringaniza mubikorwa
Gukomatanya tensor parallelism imbere ya seriveri imwe ya 8-GPU hamwe na parallelisme ihuriweho na seriveri nyinshi kugirango itoze icyitegererezo kinini cyane kumashini imwe.
Gukomatanya tensor parallelism imbere muri seriveri imwe ya 8-GPU hamwe nu muyoboro uhwanye na seriveri nyinshi kugirango uhugure icyitegererezo kinini cyane ku mashini imwe Amakipe akunze kubona umusaruro ushimishije iyo asobanuye ibipimo byujuje ubuziranenge imbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro hamwe nibiciro byamakosa mugihe runaka.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.