Sharding ak tàggat yaram buñ mëna delloosi
Pexem denc xeetu tàggat yaram ci ay piyees (shards) suko defee model yu mag yi mëna denc ak sarsewaat te duñu tënk ci memory wala disk limits, suko defee benn run bu tass mën na jël fimu bàyyeewoon ndànk.
Résumé
Essential for any training job that runs for days or weeks across many GPUs.
Plongeur bu xóot
Checkpoint tàggat yaram dafay wane lépp lu nit ñi soxla ngir mëna dellu ci liggéey bi: poid model yi, stade optimisatër yi, kalendriye taux de apprentissage, position chargeur de données, ak seeds generateur numéro aléatoire. Ci model yu mag yi, nataal bi mën na nekk téemeeri gigabyte, muy lu yaatu lool ngir benn fichier wala benn mémoire masin. Checkpoint sharding dafay xaaj nataal bi ci fichier yu bari ak rang yu bari, kon GPU bu nekk dafay bind boppam ci paralel. Resumable tàggat yaram suko defee sarsewaat yooyu shards ak defaraat stade bu mat sëkk. Su ko amul, daw ayu-bis yu bari buy daanu ci waxtu 200 dafa wara tàmbaliwaat ci noonu. Kadre yu melni PyTorch, DeepSpeed, ak xeetu safetensor yu Hugging Face Hub dañuy def lii ñuy faral di def.
Gis-gis xarala
Sharding dafay dox ndax tàggat yaram buñ séddale dafa xaaj poid yi ak réew yi gëna xéewale ci rang yi (jaaraleko ci done, tensor, wala parallelism ZeRO). Rank bu nekk dafay serialize xaaj bimu am, lu ci bari ci formaa yu melni safetensors yuy may chargement bu tayeel, buñu defaree mémoire. Fichier index dafay màndargaal tur parametre yi ci fichier shard yi. Ngir yeggaat ci anam wu leer, sistem bi dafay wéy di nekk RNG, lim jéego yi optimiser bi def, ak offset dataloader bi gëna jub, suko defee rerun bi di génne benn xeetu lots yi.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu Sharding ak tàggat yuñ mëna delloo
Checkpointing mingi joge ci xew-xew buy taxawal àdduna bi dem ci lu asynchrone te daanaka amul benn xaalis. Xaarandi yeneen ci-memoire ak checkpointing yuñ jaxasoo yuy bind ay shards ci ginaaw bi tàggat-yaram bi wéyee, boole ci checkpoints yuñ kodde efaase ak yuñ toppandoo yuy mucc ci node yu bari ci escale junni-GPU. Màngasin yi am ci niir yi ak niveau NVMe yu gëna gaaw yi dina ñu jël ay shards, ak formaa yuñ yamale yu melni safetensors dina ñu wéy di gëna yombal, gaaw, yobbu parsiel ngir tàmbaliwaat tàggat yaram ak génne inference.
Doxal ci àdduna dëgg
Benn modelu frontier dafay jaar ci ay junni GPU yuy denc ay checkpoint sharded ay téemeeri jéego yu néew yu nekk, suko defee benn node bu lajj du yàgg ay simili, du ay fan.
Hugging Face dafay séddale xeetu ubbeeku bu yaatu ni ay safetensor yu bari boole ci index.json suko defee jëfandikukat yi mëna ko yebbi ak sarse ko piyees par piyees.
Gëstukat bi dafay tàmbaliwaat ab fine-tune buñu dagg buy defaraat momentum optimiser bi, lim jéego yi, ak position dataloader bi ngir wéy di dox bu baax.
Spot-instance tàggat ci GPUs cloud yu yomb, fu ay checkpoint yu bari di tax liggéey bi mucc suñu ko dàqee ba noppi ñu defaraat ko.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Checkpoint Sharding and Resumable Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Slurm ngir kuréel yiy tàggat IA
Laaj yi ñuy faral di laaj
What is Checkpoint Sharding and Resumable Training?
Pexem denc xeetu tàggat yaram ci ay piyees (shards) suko defee model yu mag yi mëna denc ak sarsewaat te duñu tënk ci memory wala disk limits, suko defee benn run bu tass mën na jël fimu bàyyeewoon ndànk. Dafay am solo ci bépp liggéeyu tàggat yaram buy dem ay fan wala ayu-bis ci GPU yu bari.
Lu tax ñuy xaaj checkpoint yu mag yi ay pàcc?
Checkpoint yu rëy (téemeeri GB) mënu ñu nekk benn fichier; sharding dafay may rang yu bari ñu bind seeni daggite ci paralel ba noppi may ñu sarse ay piyees.
Lu weesu poids model yi, lan leneen la checkpoint buñ mëna delloosi wara denc?
Ngir tàmbaliwaat ci anam wu jaar yoon, checkpoint bi dafay denc etaa optimisatër yi, etaa generatëru nimero aleatoire yi, lim jéego yi, ak fi loader done yi bàyyee.
Lan mooy njariñ li gëna mag ci tàggat yaram ngir liggéey yu yàgg?
Ak checkpoint yuñ mëna delloosi, ab daw bu dagg dafay sarsewaat etaa bimu mujjee denc ba noppi wéy, ludul sànni ay fan ci xayma.
naka lay def ba rang GPU bu nekk di jàppale ci checkpoint buñ xaaj?
Ndax taggat buñ séddale dafa xaaj model bi ci rang yi, rang bu nekk dafay bind slice-am, loolu mooy tax sakkanal paralel ak memory-efficace.
Ban cër la fichier index di def ci checkpoint yuñ xaaj?
Benn index (lu melni, index.json) dafay bind shard bi yor parametre bu nekk suko defee chargeur yi mëna jël ak dajale piyees yi war.