Slurm don Ƙungiyoyin Horar da AI
Slurm shine mai sarrafa kayan aiki na bude-bude wanda ke tsarawa da gudanar da ayyuka akan gungu na lissafin ayyuka masu girma, kuma ya zama zaɓi na tsoho don babban horo na AI.
Dubawa
It matters because it reliably distributes massive training runs across thousands of GPUs.
Zurfafa nutsewa
Slurm (Simple Linux Utility for Resource Management) ya samo asali ne daga babban kwamfuta kuma yanzu yana iko da yawancin manyan gungu na horar da AI na duniya. Masu amfani suna ƙaddamar da rubutun tsari tare da sbatch, neman albarkatu kamar nodes da GPUs tare da umarni kamar --gres=gpu:8, da layukan Slurm, fifita fifiko, da ƙaddamar da aikin. Its srun launcher spawns coordinated processes across nodes, which pairs naturally with distributed frameworks like PyTorch DDP and NCCL. Slurm yana bin lissafin albarkatun albarkatu, yana aiwatar da daidaitaccen rabo da iyakoki, kuma yana sarrafa tsarin cikawa don sanya ƙananan ayyuka cikin giɓi. Don horar da ƙirar ƙira, ƙungiyoyi sun dogara da Slurm don sarrafa dubunnan GPUs, sake farawa daga wuraren bincike bayan gazawar kumburi, da tanadin ƙarfin sadaukarwa na tsawon makonni da yawa.
Fahimtar Fasaha
Mai sarrafa Slurm daemon (slurmctld) yana yanke shawara yayin da wakilin slurmd akan kowane kumburi yana ƙaddamar da ayyuka yana ba da rahoton matsayi. The Generic Resource (GRES) plugin yana bin GPUs don haka ayyuka suna buƙatar su a sarari. srun yana saita masu canjin yanayi (daraja, girman duniya, babban adireshin) wanda ke rarraba dakunan karatu na horo da aka karanta zuwa bootstrap sadarwar NCCL. Jadawalin cikawa na baya yana ba da damar gajerun ayyuka suyi aiki da wuri muddin ba su jinkirta tanadin fifikon fifiko ba, yana mai da amfani sosai.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Slurm don Ƙungiyoyin Koyar da AI
Slurm yana ci gaba da ƙara fashewar gajimare, goyan bayan kwantena ta hanyar Pyxis da Enroot, da madaidaitan fasalulluka na GPU. Kamar yadda gungu na AI ke da ma'auni zuwa 100,000-da GPUs, tsammanin haƙuri mai ƙarfi mai ƙarfi, haɗawa ta atomatik-sake farawa, da ayyuka masu ƙarfi waɗanda ke canzawa bayan gazawa. Ƙungiyoyi da yawa a yanzu suna gudanar da Slurm tare ko ƙarƙashin Kubernetes, kuma masu tsara tsarin tsarawa suna da nufin haɗa ingantaccen salon HPC tare da sassaucin ɗan ƙasa na gajimare don gudanar da horo mafi girma.
Aiwatar da Gaskiyar Duniya
Lab ɗin kan iyaka yana ƙaddamar da horo na makonni da yawa a cikin dubunnan GPUs tare da rubutun sbatch guda ɗaya yana buƙatar ɗaruruwan nodes.
Wani mai bincike ya ƙaddamar da 'srun --gres=gpu:8' don ɗaukar GPUs takwas akan kumburi ɗaya don gwajin PyTorch DDP.
Tsare-tsare na baya ya sanya ɗan gajeren aikin kimantawa cikin GPUs marasa aiki yayin da babban aikin horon da aka keɓe yana jiran farawa.
Bayan kumburi ya gaza tsakiyar gudu, Slurm ya nemi aikin kuma ya dawo daga sabon wurin bincike maimakon farawa.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Slurm for AI Training Clusters quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
DeepSpeed da Megatron Training Stacks
Tambayoyin da ake yawan yi
What is Slurm for AI Training Clusters?
Slurm shine mai sarrafa kayan aiki na bude-bude wanda ke tsarawa da gudanar da ayyuka akan gungu na lissafin ayyuka masu girma, kuma ya zama zaɓi na tsoho don babban horo na AI. Yana da mahimmanci saboda dogaro yana rarraba ɗimbin horo da ke gudana a cikin dubunnan GPUs.
Wane umurni ne masu amfani ke amfani da su don ƙaddamar da aikin batch zuwa Slurm?
sbatch yana ƙaddamar da rubutun tsari wanda ke bayyana albarkatun da umarni don aiki zuwa layin Slurm.
Ta yaya aikin Slurm ke buƙatar GPUs?
Tsarin Albarkatun Jini (GRES) yana ƙyale ayyuka su nemi GPUs a sarari, misali --gres=gpu:8.
Wanne bangaren Slurm ne ke yanke shawara ta tsakiya?
slurmctld shine mai sarrafa daemon wanda ke tsara ayyuka, yayin da slurmd ke gudana akan kowane kumburi don ƙaddamar da ayyuka.
Me yasa srun yana da kyau tare da tsarin horo da aka rarraba kamar PyTorch DDP?
srun yana ƙaddamar da ayyuka masu daidaitawa a cikin nodes kuma yana ba da matsayi da bayanin adireshi mai mahimmanci waɗanda ɗakunan karatu ke amfani da su don bootstrap.
Menene maƙasudin jadawalin cikawa a cikin Slurm?
Backfill yana inganta amfani ta hanyar shigar da ƙananan ayyuka cikin tsara gibi muddin ba su mayar da ayyukan da aka keɓe ba.