Nyochaa Sharding na Ọzụzụ Resumable
Usoro nke ịzọpụta a nlereanya si ọzụzụ steeti na iberibe (shards) otú ibu ụdị nwere ike ịzọpụta na reloading na-enweghị kpagbuo na ebe nchekwa ma ọ bụ disk ókè, na otú a mkpọka ọsọ nwere ike bulie kpọmkwem ebe ọ kwụsịrị.
Nchịkọta
Essential for any training job that runs for days or weeks across many GPUs.
Ime miri emi
Ebe a na-enyocha ọzụzụ bụ nseta ihuenyo nke ihe niile achọrọ iji maliteghachi: nha ihe nlere anya, steeti ndị na-ebuli elu, usoro mmụta ọnụego, ọnọdụ onye na-ebu data, na mkpụrụ osisi na-emepụta nọmba random. Maka ụdị nnukwu foto a nwere ike ịbụ narị narị gigabytes, buru oke ibu maka otu faịlụ ma ọ bụ ebe nchekwa otu igwe. Sharding checkpoint na-ekewa nke na-ese foto gafee ọtụtụ faịlụ na ọtụtụ ọkwa, yabụ GPU ọ bụla na-ede naanị ibe ya n'otu aka ahụ. Ọzụzụ nke ịmaliteghachi wee bugharịa shards ndị ahụ wee weghachi steeti zuru ezu nke ọma. Na-enweghị ya, ịgba ọsọ ọtụtụ izu nke na-akụda na elekere 200 ga-amalitegharị site na ọkọ. Frameworks dị ka PyTorch Distributed Checkpoint, DeepSpeed na Ọkpụkpọ nchekwa nchekwa nke Hugging Face Hub na-eme usoro a.
Nghọta nka nka
Sharding na-arụ ọrụ n'ihi na ọzụzụ ekesara ekewalarị nha nha yana steeti kachasị mma n'ofe ọkwa (site na data, tensor, ma ọ bụ ZeRO). Ọkwa ọ bụla na-esetịpụ naanị nkebi ya, na-emekarị ka ọ bụrụ usoro dị ka safetensors nke na-ekwe ka ndị umengwụ na-ebu ibu, ebe nchekwa mapụtara. Otu faịlụ ndeksi na-edepụta oke aha faịlụ shard. Iji maliteghachi n'ụzọ doro anya, sistemụ ahụ na-aga n'ihu na steeti RNG, ọnụ ọgụgụ nzọụkwụ kachasị mma, yana nkwụsị nke dataloader, yabụ nrụgharị ahụ na-emegharị otu usoro nke batches.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Sharding Checkpoint na Ọzụzụ Resumable
Ntụle nlele na-agbanwe site na mmemme nkwụsị ụwa na-eme kwa oge gaa na ihe na-ekwekọghị ekwekọ na ọ fọrọ nke nta ka ọ bụrụ n'efu. Na-atụ anya n'ihe ncheta na nlebanya kpuchiri ekpuchi nke na-ede njakịrị n'azụ ka ọzụzụ na-aga n'ihu, gbakwunyere ihichapụ koodu na ebe nlele emegharịgharị nke na-adị ndụ ọdịda ọnụ ọnụ na-ahụkarị na nha puku GPU. Ụlọ ahịa ihe igwe ojii na ọkwa NVMe mpaghara ngwa ngwa ga-anabata shards, na usoro ahaziri ahazi dị ka safetensors ga-aga n'ihu na-emeziwanye nchekwa, ngwa ngwa, ntinye akụkụ maka ma ịmaliteghachi ọzụzụ yana ntinye ntinye.
Mmejuputa n'ezie n'ụwa
Ihe nlere anya na-agba ọsọ gafee puku kwuru puku GPU nke na-echekwa akpaghị aka na-echekwa ebe a na-enyocha ihe nrịbama ọ bụla usoro narị ole na ole ka otu oghere dara ada na-efu naanị nkeji, ọ bụghị ụbọchị.
Ihu ịmakụ na-ekesa nnukwu ụdị mepere emepe dị ka ọtụtụ shards safetensors gbakwunyere index.json ka ndị ọrụ nwee ike budata ma buru ya n'otu ibe.
Onye nyocha na-amalitegharị mma-akụ akwụsịre nke na-eweghachite ngwa ngwa ngwa ngwa, ọnụọgụ nzọụkwụ na ọnọdụ nbudata data ka ọ gaa n'ihu n'enweghị nsogbu.
Ọzụzụ ihe atụ na GPU igwe ojii na-adịghị adị ọnụ ala, ebe ebe a na-enyocha ya ugboro ugboro na-ahapụ ọrụ ahụ ka a na-achụpụ ya na ịhazigharị ya.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Checkpoint Sharding and Resumable Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Slurm maka ụyọkọ Ọzụzụ AI
Ajụjụ a na-ajụkarị
What is Checkpoint Sharding and Resumable Training?
Usoro nke ịzọpụta a nlereanya si ọzụzụ steeti na iberibe (shards) otú ibu ụdị nwere ike ịzọpụta na reloading na-enweghị kpagbuo na ebe nchekwa ma ọ bụ disk ókè, na otú a mkpọka ọsọ nwere ike bulie kpọmkwem ebe ọ kwụsịrị. Dị mkpa maka ọrụ ọzụzụ ọ bụla nke na-aga ụbọchị ma ọ bụ izu n'ofe ọtụtụ GPU.
Gịnị mere e ji kewaa ebe a na-enyocha ihe nleba anya ka ọ bụrụ nza?
Nnukwu ebe nlele (ọtụtụ narị GB) adịghị arụ ọrụ dị ka otu faịlụ; Sharding na-ahapụ ọtụtụ ọkwa ka ha dee ibe ha n'otu n'otu ma na-enyere aka ịkwanye ibe ya.
E wezụga ịdị arọ ihe atụ, olee ihe ọzọ ebe nleba anya ga-amaliteghachi na-echekwa?
Iji malitekwa nke ọma, ebe nleba anya na-echekwa steeti ndị na-ebupụta ihe, steeti-ọnụọgụ-generator, ọnụọgụ nzọụkwụ, na ebe nchekwa data kwụsịrị.
Kedu uru bụ isi nke ọzụzụ ịmaliteghachi maka ogologo ọrụ?
Site na nlegharị anya nlegharị anya, ọsọ kwụsịrị na-ebughachi ọnọdụ echekwara ikpeazụ ya wee gaa n'ihu, kama ịtụfu ụbọchị mgbakọ.
Kedu ka ọkwa GPU nke ọ bụla na-esi enyekarị aka na ebe nleba anya nzacha?
N'ihi na ọzụzụ kesara ekewapụla ihe nlereanya n'ofe ọkwa, ọkwa nke ọ bụla na-ede naanị ibe ya, na-eme ka nchekwa ya dị ka nke ọma na nke ọma.
Kedu ọrụ faịlụ ndeksi na-arụ na ebe a na-enyocha ihe ndị a kpara akpa?
Ndekọ ndeksi (dịka ọmụmaatụ, index.json) na-edekọ nke shard na-ejide oke nke ọ bụla ka ndị na-ebu ibu nwee ike iweta ma chịkọtakwa akụkụ ziri ezi.