Byuzuye Byuzuye Ibyatanzwe
Byuzuye Byuzuye Data Parallel (FSDP) nubuhanga bukwirakwizwa bwamahugurwa agabanya ibipimo byikitegererezo, gradients, hamwe na optimizer leta muri GPU nyinshi kuburyo buri gikoresho gifata igice gusa.
Incamake
It makes training huge models possible on hardware that could never fit the whole model in one GPU's memory.
Kwibira cyane
Imibare gakondo ibangikanye ibika kopi yuzuye yicyitegererezo kuri buri GPU, isesagura ububiko na caps ingano yubunini. FSDP, ikunzwe na __AIU_PROTECTED_13 __ 'PyTorch kandi ihumekwa na ZeRO ya __AIU_PROTECTED_5 __, ahubwo igabanya ibintu bitatu mubikoresho: ibipimo, gradients, hamwe na leta nziza. Mugihe cyo gutambuka imbere, buri GPU ikusanya by'agateganyo uburemere bwuzuye kurwego irimo kubara ikoresheje byose-ikusanya, ikora ibarwa, hanyuma ihita irekura kopi yakusanyije. Inyuma yinyuma ikora kimwe, ikurikirwa no kugabanya-gutatanya gukwirakwiza uduce duto duto dusubira kuri GPU zabo. Kuberako buri gikoresho kibika burundu igice cyicyitegererezo, gukoresha kwibuka bitonyanga hafi yumubare wa GPUs, ukareka amakipe akamenyereza moderi hamwe na miliyari icumi cyangwa magana.
Ubushishozi
FSDP icuruza itumanaho ryiyongera kubitsa. Ibipimo bya buri cyiciro byongeye kubakwa kubisabwa hamwe byose hamwe mbere yo gukoreshwa hanyuma bikajugunywa nyuma, mugihe gradients zahujwe kandi zigabanijwe kugabanya-gutatanya. Itumanaho rirashobora guhuzagurika no kubara muguhitamo ibipimo bikurikiraho mugihe urwego rwubu rukora, guhisha byinshi mubukererwe bwurusobe. Kuringaniza granularity (politike yo gupfunyika) iringaniza ikirenge cyo kwibuka hamwe n'itumanaho hejuru.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Ejo hazaza huzuye amakuru asanganywe
FSDP ihinduka nkibisanzwe byugururiwe amahugurwa manini-manini, hamwe na FSDP2 muri PyTorch itezimbere imikoreshereze hamwe na buri kintu cyerekana. Witegereze kurushaho guhuza hamwe na tensor hamwe nuyoboro ugereranije kubintu bya tiriyari-ibipimo byerekana, inkunga nziza yo kuvanga neza na fp8, hamwe no gufunga ubwenge byikora bitoranya imipaka kuri wewe. Mugihe imiyoboro ya GPU ihuza nka NVLink na InfiniBand igenda yihuta, igiciro cyitumanaho rya sharding gikomeza kugabanuka, bigatuma gifatika mubipimo binini cyane.
Gushyira mu bikorwa Isi
Kuringaniza neza miliyari 70-yerekana ibipimo bya Llama muri 8 GPUs kugiti cye kidashobora gufata uburemere bwuzuye.
Gutegura imvugo nini yindimi muri laboratoire ya AI mugukata ibintu byiza (biganje mububiko hamwe na Adam) mumajana yihuta.
Abashakashatsi bifashishije igikoresho cya FSDP cya PyTorch kugirango bahugure abahindura icyerekezo kuri cluster ya kaminuza bataguze 80GB GPUs.
Gukomatanya FSDP hamwe na mix-precision bfloat16 kugirango ugabanye hafi kwibuka no kwihutisha amahugurwa kuri moderi nyinshi.
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Fully Sharded Data Parallel quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kuringaniza amakuru
Ibibazo bikunze kubazwa
What is Fully Sharded Data Parallel?
Byuzuye Byuzuye Data Parallel (FSDP) nubuhanga bukwirakwizwa bwamahugurwa agabanya ibipimo byikitegererezo, gradients, hamwe na optimizer leta muri GPU nyinshi kuburyo buri gikoresho gifata igice gusa. Bituma imyitozo nini nini ishoboka kubikoresho bidashobora na rimwe guhuza moderi yose mububiko bumwe bwa GPU.
Niki FSDP igabanya muri GPUs ko amakuru asanzwe adahuye?
FSDP igabanya ibipimo by'icyitegererezo, gradients, hamwe na optimizer leta mubikoresho, mugihe imibare isanzwe ibangikanye yigana icyitegererezo cyuzuye kuri buri GPU.
Nibihe bikorwa rusange FSDP ikoresha kugirango yubake urwego rwuzuye mbere yo kubara?
Mbere yuko layer ikora, FSDP ikora byose-guteranya kugirango ikusanyirize by'agateganyo ibipimo byuzuye uhereye kumashanyarazi yose, hanyuma ikabohora nyuma.
Kuki FSDP irekura ibipimo byuzuye byegeranijwe ako kanya nyuma yo kubara?
Gufata ikariso gusa burundu no kwegeranya ibipimo byuzuye mugihe gito nibyo bituma imikoreshereze yibuke iba mike kandi igereranijwe nigice kimwe cyicyitegererezo.
FSDP yahumekewe cyane nuburyo bwambere bwo kwibuka-optimizas?
FSDP igabanya ibipimo, gradients, hamwe na optimizer ikurikirana neza ibitekerezo byatangijwe muri __AIU_PROTECTED_5 __ 'ZeRO kuva mubitabo byimbitse.
Nigute FSDP ihisha byinshi mubukererwe bwurusobe gukusanya ibiro?
FSDP ibanziriza ibipimo bikurikiraho mugihe urwego rwubu ruracyabaze, rwuzuza itumanaho ryose hamwe nakazi keza.