Sharding Checkpoint da Ci gaba da Horarwa
Dabaru don adana yanayin horon samfurin guntu (shards) don haka za a iya ajiye manyan samfura kuma a sake loda su ba tare da shaƙawa akan iyakokin ƙwaƙwalwar ajiya ko faifai ba, don haka gudu ya faɗo zai iya ɗauka daidai inda ya tsaya.
Dubawa
Essential for any training job that runs for days or weeks across many GPUs.
Zurfafa nutsewa
Wurin binciken horo hoto ne na duk abin da ake buƙata don ci gaba: ma'aunin ƙira, jahohin ingantawa, jadawalin ƙimar koyo, matsayin mai ɗaukar bayanai, da nau'in janareta na lambar bazuwar. Ga manyan samfura wannan hoton na iya zama ɗaruruwan gigabytes, wanda yayi girma da yawa ga fayil ɗaya ko ƙwaƙwalwar na'ura ɗaya. Sharding Checkpoint yana rarrabuwa wanda ke ɗaukar hoto a cikin fayiloli da yawa da darajoji da yawa, don haka kowane GPU yana rubuta nasa yanki kawai a layi daya. Horon da za a sake dawowa sannan ya sake loda waɗancan shards kuma ya dawo da cikakken jihar daidai. Idan ba tare da shi ba, gudu na makonni da yawa wanda ya fado a sa'a 200 zai sake farawa daga karce. Tsarukan kamar PyTorch Rarraba Checkpoint, DeepSpeed , da Tsarin Rarraba Face Hub sun sanya wannan na yau da kullun.
Fahimtar Fasaha
Sharding yana aiki saboda horon da aka rarraba ya riga ya raba ma'aunin nauyi da jihohi masu ingantawa a cikin matsayi (ta hanyar bayanai, tensor, ko daidaitaccen ZeRO). Kowane matsayi yana tsara ɓangaren sa ne kawai, sau da yawa zuwa tsari kamar safetensors waɗanda ke ba da damar raɗaɗi, ɗaukar taswirar ƙwaƙwalwar ajiya. Fayil na fihirisa taswirar siga sunaye zuwa fayilolin shard. Don ci gaba da ƙayyadaddun ƙayyadaddun ƙayyadaddun ƙayyadaddun tsarin, tsarin kuma yana dawwama a jihohin RNG, ƙidayar matakin ingantawa, da daidaitaccen mai ɗaukar bayanai, don haka sake kunnawa yana sake fitar da jerin batches iri ɗaya.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Sharding Point Point da Horowar da za a Ci gaba
Dubawa yana jujjuya daga taron na lokaci-lokaci na duniya zuwa wani abu mara daidaituwa kuma kusan kyauta. Yi tsammanin ƙarin abin tunawa da madaidaicin ma'aunin bincike wanda ke rubuta shards a bango yayin da ake ci gaba da horarwa, tare da share-coded da maimaita wuraren bincike waɗanda ke tsira da gazawar kumburi gama gari a sikelin-GPU dubu. Shagunan abubuwan girgije da matakan NVMe na gida masu sauri za su dauki nauyin shards, kuma daidaitattun tsari kamar safetensors za su ci gaba da inganta aminci, da sauri, ɗaukar juzu'i don duka dawo da horo da ƙaddamar da ƙaddamarwa.
Aiwatar da Gaskiyar Duniya
Samfurin iyaka yana gudana a cikin dubunnan GPUs waɗanda ke adana wuraren bincike ta atomatik kowane ƴan matakai ɗari don haka kumburin da ya gaza ɗaya kawai yana ɗaukar mintuna, ba kwanaki ba.
Fuskar Runguma tana rarraba babban buɗaɗɗen ƙira a matsayin shards na safetensors da yawa tare da index.json don masu amfani su iya saukewa da loda shi gabaɗaya.
Wani mai bincike yana ci gaba da katsewa mai kyau wanda ke dawo da ainihin lokacin ingantawa, ƙidayar mataki, da matsayin mai ɗaukar bayanai don ci gaba da watsewa.
Horarwa ta misali akan GPUs masu arha mai arha, inda wuraren bincike akai-akai ke barin aikin ya tsira daga korar da sake tsarawa.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Checkpoint Sharding and Resumable Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Slurm don Ƙungiyoyin Horar da AI
Tambayoyin da ake yawan yi
What is Checkpoint Sharding and Resumable Training?
Dabaru don adana yanayin horon samfurin guntu (shards) don haka za a iya ajiye manyan samfura kuma a sake loda su ba tare da shaƙawa akan iyakokin ƙwaƙwalwar ajiya ko faifai ba, don haka gudu ya faɗo zai iya ɗauka daidai inda ya tsaya. Mahimmanci ga kowane aikin horo da ke gudana na kwanaki ko makonni a cikin GPUs da yawa.
Me yasa manyan wuraren bincike na samfuri suka rabu zuwa shards?
Manyan wuraren bincike (daruruwan GB) ba su da amfani a matsayin fayil ɗaya; Sharding yana ba da dama da yawa su rubuta yankan su a layi daya kuma yana ba da damar yin lodawa gabaɗaya.
Bayan ma'aunin ƙira, menene kuma dole ne madaidaicin wurin bincike yakan adana?
Don ci gaba daidai, wurin binciken yana adana jihohin ingantawa, jihohin janareta bazuwar, ƙidayar mataki, da inda mai ɗaukar bayanai ya tsaya.
Menene babban fa'idar sake dawowa horo don dogon ayyuka?
Tare da ci gaba da wuraren bincike, guduwar da aka katse ta sake loda yanayin sa na ƙarshe kuma ya ci gaba, maimakon zubar da kwanakin ƙididdiga.
Ta yaya kowane matsayi na GPU yakan ba da gudummawa zuwa wurin bincike mai faɗuwa?
Saboda horon da aka rarraba ya riga ya raba samfurin a cikin darajoji, kowane matsayi yana rubuta yanki ne kawai, yana yin daidaici da ingantaccen ƙwaƙwalwar ajiya.
Wace rawa fayil ɗin fihirisa ke takawa a wuraren bincike da aka shaƙa?
Fihirisa (misali, index.json) yana yin rikodin abin da shard ke riƙe kowane siga ta yadda masu ɗaukar kaya za su iya ɗauko su sake haɗa daidaitattun guda.