Incamake
Data parallelism itoza icyitegererezo cyihuse mukwigana muri GPU nyinshi, hamwe na GPU itunganya ibice bitandukanye byamakuru. Nubuhanga bwakazi bukora butuma amakipe apima kugera kubihumbi cyangwa ibihumbi byihuta.
Data Parallelism ni inyubako ya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa kurwego.
Kwibira cyane
Muburinganire bwamakuru, buri GPU ifite kopi imwe yuburemere bwikitegererezo ariko itunganya mini-batike yingero zamahugurwa. Buri gikoresho kibara imbere ninyuma yinyuma yigenga, itanga umurongo wa gradients. Mbere yo kuvugurura ibipimo, gradients zigereranijwe muri GPU zose ukoresheje ibikorwa byose bigabanya itumanaho, bityo buri kopi iguma muri sync kandi ikitwara nkaho yatojwe kumurongo umwe uhuriweho. Ibi bigwiza neza ibyinjira: 8 GPU irashobora guhekenya hafi 8x amakuru kuri buri ntambwe. Gufata ni uko buri GPU igomba guhuza icyitegererezo cyose, gradients zayo, hamwe na optimizer leta murwibutso, kubwibyo rero amakuru asobanutse ntagufasha mugihe icyitegererezo ari kinini kubikoresho bimwe.
Ubushishozi
Igikorwa cyingenzi ni byose-bigabanya, bikusanya gradients hejuru yibikoresho kandi bigabura ibisubizo. Impeta yose-igabanya, ikoreshwa namasomero nka NCCL na Horovod, itambutsa uduce duto duto tuzengurutse impeta yumvikana kuburyo itumanaho ryose ritigenga kubara GPU. PyTorch's DistributedDataParallel ihuza itumanaho hamwe ninyuma yinyuma, ikuraho sync ya sync ya gradient kubice byambere mugihe nyuma ibice biracyabazwe, bihisha byinshi mubukererwe bwurusobe.
Kumenya amakuru Kubangikanya
Data parallelism itoza icyitegererezo cyihuse mukwigana muri GPU nyinshi, hamwe na GPU itunganya ibice bitandukanye byamakuru. Nubuhanga bwakazi bukora butuma amakipe apima kugera kubihumbi cyangwa ibihumbi byihuta. Data Parallelism ni inyubako ya tekiniki yubaka igira ingaruka kumiterere yicyitegererezo, igiciro cyibikorwa remezo, ubukererwe, no kwizerwa kurwego. Kugirango wubake byimbitse, fata Data Parallelism nkicyitegererezo gikora, ntabwo ari ikintu kimwe: gusobanura ibyagezweho, gusobanura ibitekerezo, no gutandukanya ibyo sisitemu ishobora gukora byizewe nibisaba guca imanza zinzobere.
Mu myitozo, amatsinda akomeye akoresha Data Parallelism atezimbere ubwubatsi, amakuru, hamwe nibikorwa remezo bihitamo kwizerwa nigiciro. Bandika ibipimo ngenderwaho byerekana intsinzi, bagerageza kurwanya amakuru afatika hamwe nakazi keza, kandi bagasubiramo bashingiye kubikorwa byagaragaye ko batsinzwe aho gutsinda inshuro imwe. Aha niho imyumvire yubumenyi ihinduka mubushobozi burambye kubicuruzwa, politiki, nibikorwa.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mugihe kimwe, Guhindura igipimo kimwe gishobora guhisha intege nke za sisitemu. Uburyo bukomeye cyane ni uguhuza umuvuduko wikigereranyo hamwe na disipuline yimiyoborere: kuyobora abaderevu, gufata ibimenyetso, gutangaza ibyemezo byicyemezo, no gukomeza kuvugurura uburyo bwo kwirinda nkimyitwarire yicyitegererezo, ibyo abakoresha bategereje, nibisabwa n'amategeko bigenda bihinduka.
Ingaruka z'Ingamba
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa. Mubikorwa byujuje ubuziranenge, ibi bihindurwa mumategeko agenga imikorere, imipaka nyirubwite, hamwe n'imihango yo gusubiramo kenshi kugirango amakipe ashobore kwigirira ikizere aho gupima ibidasobanutse.
Gushyira mu bikorwa Isi
Guhugura ResNet ishusho itondekanya muri GPU 8 muri seriveri imwe ukoresheje PyTorch IkwirakwizwaDataParallel, buri GPU ikora 32 muri 256-ishusho.
Gupima BERT kwitegereza amagana ya GPU hamwe na Horovod, ukoresheje impeta byose-kugabanya kugirango uhuze gradients buri ntambwe.
Kuringaniza neza icyitegererezo cyicyitegererezo kuri cluster-node aho buri node itunganya abakoresha-imikoranire itandukanye.
Koresha MirroredStrategy ya TensorFlow kugirango ukwirakwize amahugurwa yicyitegererezo muri GPU nyinshi kumurongo umwe wakazi hamwe nimpinduka ntoya.
Uburyo bwo Gushyira mu bikorwa
Guhuza amakuru mubikorwa
Guhugura ResNet ishusho itondekanya muri GPU 8 muri seriveri imwe ukoresheje PyTorch IkwirakwizwaDataParallel, buri GPU ikora 32 muri 256-ishusho.
Guhugura ResNet ishusho itondekanya muri GPUs 8 muri seriveri imwe ukoresheje PyTorch IkwirakwizwaDataParallel, buri GPU ikora 32 muri 256-ishusho yitsinda ryamakipe ubusanzwe ibona ibisubizo byiza mugihe isobanura ibipimo byujuje ubuziranenge imbere, igakomeza inzira yo kuzamuka kwabantu kubibazo bikurikirana, kandi ikurikirana inyungu zibyara umusaruro nibiciro byigihe.
Guhuza amakuru mubikorwa
Gupima BERT kwitegereza amagana ya GPU hamwe na Horovod, ukoresheje impeta byose-kugabanya kugirango uhuze gradients buri ntambwe.
Gupima BERT yerekana amajana n'amajana ya GPU hamwe na Horovod, ukoresheje impeta zose-kugabanya kugirango uhuze gradients buri ntambwe Amakipe akunze kubona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Guhuza amakuru mubikorwa
Kuringaniza neza icyitegererezo cyicyitegererezo kuri cluster-node aho buri node itunganya abakoresha-imikoranire itandukanye.
Guhuza neza icyitegererezo cyicyifuzo kuri cluster-node aho buri node itunganya abakoresha-imikoranire itandukanye Amakipe ubusanzwe abona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere imbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byamakosa mugihe runaka.
Guhuza amakuru mubikorwa
Koresha MirroredStrategy ya TensorFlow kugirango ukwirakwize amahugurwa yicyitegererezo muri GPU nyinshi kumurongo umwe wakazi hamwe nimpinduka ntoya.
Koresha MirroredStrategy ya TensorFlow kugirango ukwirakwize amahugurwa yicyitegererezo muri GPU nyinshi kumurongo umwe wakazi hamwe nimpinduka ntoya ya kipe Amakipe akunze kubona ibisubizo byiza iyo asobanuye ibipimo ngenderwaho byimbere, agakomeza inzira yo kuzamura abantu kubibazo byimbere, kandi akurikirana inyungu zibyara umusaruro nibiciro byigihe.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima. Fata buri ntambwe nk irembo ryibimenyetso: niba ibipimo bitujujwe, hagarika kuzenguruka, funga icyuho, hanyuma noneho wagure imikoreshereze.