Ubuyobozi bwa tekiniki
Byuzuye Byuzuye Ibyatanzwe
Byuzuye Byuzuye Data Parallel (FSDP) nubuhanga bukwirakwizwa bwamahugurwa agabanya ibipimo byikitegererezo, gradients, hamwe na optimizer leta muri GPU nyinshi kuburyo buri gikoresho gifata igice gusa.
Umva iki gitaboAI-byavuzwe · ~2 min
Kuriyi page2 min soma
Incamake
Bituma imyitozo nini ishoboka kubikoresho bidashobora na rimwe guhuza moderi yose mububiko bumwe bwa GPU.
Kwibira cyane
Imibare gakondo ibangikanye ibika kopi yuzuye yicyitegererezo kuri buri GPU, isesagura ububiko na caps ingano yubunini. FSDP, ikunzwe na __AIU_PROTECTED_13 __ 'PyTorch kandi ihumekwa na ZeRO ya __AIU_PROTECTED_5 __, ahubwo igabanya ibintu bitatu mubikoresho: ibipimo, gradients, hamwe na leta nziza. Mugihe cyo gutambuka imbere, buri GPU ikusanya by'agateganyo uburemere bwuzuye kurwego irimo kubara ikoresheje byose-ikusanya, ikora ibarwa, hanyuma ihita irekura kopi yakusanyije. Inyuma yinyuma ikora kimwe, ikurikirwa no kugabanya-gutatanya gukwirakwiza uduce duto duto dusubira kuri GPU zabo. Kuberako buri gikoresho kibika burundu igice cyicyitegererezo, gukoresha kwibuka bitonyanga hafi yumubare wa GPUs, ukareka amakipe akamenyereza moderi hamwe na miliyari icumi cyangwa magana.
Gushyira mu bikorwa Isi
Kuringaniza neza miliyari 70-yerekana ibipimo bya Llama muri 8 GPUs kugiti cye kidashobora gufata uburemere bwuzuye.
Gutegura imvugo nini yindimi muri laboratoire ya AI mugukata ibintu byiza (biganje mububiko hamwe na Adam) mumajana yihuta.
Abashakashatsi bifashishije igikoresho cya FSDP cya PyTorch kugirango bahugure abahindura icyerekezo kuri cluster ya kaminuza bataguze 80GB GPUs.
Gukomatanya FSDP hamwe na mix-precision bfloat16 kugirango ugabanye hafi kwibuka no kwihutisha amahugurwa kuri moderi nyinshi.
Ejo hazaza huzuye amakuru asanganywe
FSDP ihinduka nkibisanzwe byugururiwe amahugurwa manini-manini, hamwe na FSDP2 muri PyTorch itezimbere imikoreshereze hamwe na buri kintu cyerekana. Witegereze kurushaho guhuza hamwe na tensor hamwe nuyoboro ugereranije kubintu bya tiriyari-ibipimo byerekana, inkunga nziza yo kuvanga neza na fp8, hamwe no gufunga ubwenge byikora bitoranya imipaka kuri wewe. Mugihe imiyoboro ya GPU ihuza nka NVLink na InfiniBand igenda yihuta, igiciro cyitumanaho rya sharding gikomeza kugabanuka, bigatuma gifatika mubipimo binini cyane.
Kubishyira mubikorwa
Impamvu ari ngombwa
- Igiciro na bije. Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
- Ibyemezo bisobanutse. Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
- Kugenzura ubuziranenge. Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Witondere
- Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
- Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
- Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Intambwe ikurikira
- Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
- Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
- Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
- Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Ibaruwa y'amakuru y'ubuntu
Ibuka amakuru ya buri munsi ya AI
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Take the Fully Sharded Data Parallel quiz
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Ibibazo bikunze kubazwa
Niki Cyuzuye Cyuzuye Data Iringaniza?
Byuzuye Byuzuye Data Parallel (FSDP) nubuhanga bukwirakwizwa bwamahugurwa agabanya ibipimo byikitegererezo, gradients, hamwe na optimizer leta muri GPU nyinshi kuburyo buri gikoresho gifata igice gusa. Bituma imyitozo nini nini ishoboka kubikoresho bidashobora na rimwe guhuza moderi yose mububiko bumwe bwa GPU.
Niki FSDP igabanya muri GPUs ko amakuru asanzwe adahuye?
FSDP igabanya ibipimo by'icyitegererezo, gradients, hamwe na optimizer leta mubikoresho, mugihe imibare isanzwe ibangikanye yigana icyitegererezo cyuzuye kuri buri GPU.
Nibihe bikorwa rusange FSDP ikoresha kugirango yubake urwego rwuzuye mbere yo kubara?
Mbere yuko layer ikora, FSDP ikora byose-guteranya kugirango ikusanyirize by'agateganyo ibipimo byuzuye uhereye kumashanyarazi yose, hanyuma ikabohora nyuma.
Kuki FSDP irekura ibipimo byuzuye byegeranijwe ako kanya nyuma yo kubara?
Gufata ikariso gusa burundu no kwegeranya ibipimo byuzuye mugihe gito nibyo bituma imikoreshereze yibuke iba mike kandi igereranijwe nigice kimwe cyicyitegererezo.
FSDP yahumekewe cyane nuburyo bwambere bwo kwibuka-optimizas?
FSDP igabanya ibipimo, gradients, hamwe na optimizer ikurikirana neza ibitekerezo byatangijwe muri __AIU_PROTECTED_5 __ 'ZeRO kuva mubitabo byimbitse.
Nigute FSDP ihisha byinshi mubukererwe bwurusobe gukusanya ibiro?
FSDP ibanziriza ibipimo bikurikiraho mugihe urwego rwubu ruracyabaze, rwuzuza itumanaho ryose hamwe nakazi keza.
Komeza wige
Kuyobora
Abandi bayobozi batoranijwe kuriyi ngingo