Ndekota data zuru oke
Ngwakọta Data Sharded zuru oke (FSDP) bụ usoro ọzụzụ ekesa ekesa nke na-ekewa parampat ihe atụ, gradients, na steeti njikarịcha n'ofe ọtụtụ GPU ka ngwaọrụ ọ bụla na-ejide naanị iberi.
Nchịkọta
It makes training huge models possible on hardware that could never fit the whole model in one GPU's memory.
Ime miri emi
Myirịta data ọdịnala na-edobe nnomi zuru oke nke ihe nlereanya na GPU ọ bụla, nke na-emebi ebe nchekwa yana nha nha ụdị. FSDP, nke __AIU_PROTECTED_13_'s PyTorch na-ewu ewu na nke Microsoft's ZeRO, kama na-akụrisị ihe atọ n'ofe ngwaọrụ: paramita, gradients, na steeti nkwalite. N'oge ngafe na-aga n'ihu, GPU nke ọ bụla na-achịkọta ọnụ ọgụgụ zuru ezu maka oyi akwa ọ na-agbakọ site na mkpokọta, na-agbakọ mkpokọta ahụ, wee wepụ akwụkwọ a gbakọtara ozugbo. Ngafe azụ azụ na-arụ ọrụ n'otu aka ahụ, na-esote mbelata-achụsasị nke na-ekesa mpekere gradient azụ na GPU nke ha nwere. N'ihi na ngwaọrụ ọ bụla na-echekwa obere ihe nlereanya ahụ kpamkpam, iji ebe nchekwa na-adaba n'ụzọ ziri ezi na ọnụ ọgụgụ GPU, na-ahapụ ndị otu ịzụ ụdị nwere iri iri ma ọ bụ narị ijeri paramita.
Nghọta nka nka
FSDP na-azụta nzikọrịta ozi maka nchekwa nchekwa. A na-arụzigharị ihe dị arọ nke oyi akwa ọ bụla na-achọ ya na-agbakọta ozugbo tupu ejiri ya wee tụfuo ya ozugbo, ebe a na-ejikọta gradients ma kewaa ya na mbelata-gbasasịa. Enwere ike iji kọmpụta mechie nkwurịta okwu site n'ịmepụta paramita oyi akwa na-esote mgbe oyi akwa dị ugbu a na-agba ọsọ, na-ezobe ọtụtụ nkwụsị netwọkụ. Idozi granularity sharding (atụmatụ mkpuchi) na-edozi akara ukwu ebe nchekwa megide n'elu nkwukọrịta.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke ndakọrịta data zuru oke
FSDP na-aghọ ihe ndabara maka ọzụzụ nnukwu ihe nlere anya, yana FSDP2 na PyTorch na-emeziwanye ojiji yana sharding kwa-parameter. Na-atụ anya njikọta siri ike na tensor na pipeline parallelism maka ụdị trillion-parameter, nkwado ka mma maka izizi agwakọta na fp8, yana mkpuchi akpaaka kacha mma nke na-ahọrọ gị oke. Dị ka inter-GPU njikọ dị ka NVLink na InfiniBand na-enweta ngwa ngwa, ọnụ ahịa nzikọrịta ozi nke sharding na-ebelata, na-eme ka ọ dị irè n'ọtụtụ buru ibu.
Mmejuputa n'ezie n'ụwa
Ndozi nke ọma ụdị Llama nha ijeri iri asaa n'ofe 8 GPU nke n'otu n'otu enweghị ike ijide oke zuru oke.
Ịmalite ụdị asụsụ buru ibu na ụlọ nyocha AI site n'ịkwado steeti optimizer (nke na-achịkwa ebe nchekwa Adam) gafere ọtụtụ narị ngwa ngwa.
Ndị na-eme nchọpụta na-eji ihe mkpuchi FSDP PyTorch na-azụ ihe ntụgharị ọhụụ na ụyọkọ mahadum na-azụtaghị flagship 80GB GPUs.
Na-ejikọta FSDP na bfloat16 agwakọta-nkenke ka o welata ebe nchekwa ma mee ka mmepụta ọzụzụ dị ngwa na ụdị multimodal.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Fully Sharded Data Parallel quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Ndekota data
Ajụjụ a na-ajụkarị
What is Fully Sharded Data Parallel?
Ngwakọta Data Sharded zuru oke (FSDP) bụ usoro ọzụzụ ekesa ekesara nke na-ekewa parampat ihe atụ, gradients, na steeti njikarịcha n'ofe ọtụtụ GPU ka ngwaọrụ ọ bụla na-ejide naanị iberi. Ọ na-eme ka ọzụzụ dị ukwuu nwee ụdị ngwaike na-enweghị ike dabara n'ụdị niile na ebe nchekwa GPU.
Kedu ihe FSDP shard gafee GPU nke ọkọlọtọ data myirịta na-adịghị?
FSDP na-akụda paramita ihe atụ, gradients, na njikarịcha na-ekwu n'ofe ngwaọrụ, ebe data ọkọlọtọ na-egosipụta ụdị zuru oke na GPU ọ bụla.
Kedu ọrụ mkpokọta FSDP na-eji wughachi oke nha nke oyi akwa tupu ịgbakọ ya?
Tupu oyi akwa agbaa, FSDP na-eme mkpọkọta niile iji chịkọta parampat zuru ezu na shards niile, wee hapụ ha ma emechaa.
Kedu ihe kpatara FSDP ji na-atọhapụ ihe ọ̀tụ̀tụ̀ a chịkọtara n'uju ozugbo agbakọchara oyi akwa?
Ijide naanị shard na-adịgide adịgide na ịnakọta oke n'uju nwa oge bụ ihe na-eme ka ojiji ebe nchekwa dị ala na nke dabara na otu akụkụ nke ihe nlereanya ahụ.
FSDP sitere n'ike mmụọ nsọ nke ukwuu site na ụzọ nkwalite-ebuli ebe nchekwa mbụ?
FSDP's Sharding nke parameters, gradients, and optimizer states na-esochi echiche ndị ewepụtara na Microsoft's ZeRO site na ọba akwụkwọ DeepSpeed .
Kedu ka FSDP si ezobe ọtụtụ nkwụsị netwọkụ site n'ịchịkọta ibu?
FSDP na-ebupụta paramita oyi akwa nke na-esote ka oyi akwa dị ugbu a ka na-agbakọ, na-ejikọta nzikọrịta ozi niile na ọrụ bara uru.