Saytu degrade
Checkpointing gradient (ñu koy woowe itam checkpointing activation) ab pexe la buy sakkanal mémoire biy sànni activation yu bari yi ci digg bi ci diir bi ñuy jaar ci kanam, ba noppi di leen xaymaat ci diir bi ñuy tasaare ci ginaaw.
Résumé
It lets you train deeper, larger networks by trading extra compute for much lower memory use.
Plongeur bu xóot
Taggat reso neuronal yi dañuy denc bépp couche buy tàmbali ci diiru paas bi ci kanam ndax backpropagation daf leen soxla ngir xayma gradient yi. Ci model yu xóot yi, liggéey yooyu ñooy ëpp doole ci mémoire bi. Lu moy loolu, checkpoint gradient dafay denc aktivaasioŋ yi ci ay 'checkpoint' yu néew, ba noppi sànni leneen li. Su backprop yeggee ci gox buñu bàyyi activation yi, dafay defaraat calcul bi ci kanam ngir segment boobu rek ngir defaraat limu soxla, ba noppi dem. Ak checkpoints yuñ def ci lu tollu ci bepp couche square-de-N, mémoire ngir activations dafay wàcci ci komànd N dem ci komànd square-root-of-N, ci noonu la calcul di yokk lu tollu ci benn pass forward (ci diggante 20-30% gëna néew). Loolu dafay tax ñu mëna jëfandikoo ay batch yu gëna mag wala ay transformatër yu gëna xóot ci benn GPU.
Gis-gis xarala
Pexem dafay jëfandikoo kompromis diggante waxtu ak fàttaliku. Denc bépp aktivaasioŋ lu gaaw la waaye dafay xiif ci mémoire; defaraat leen lu yomb la ci accelerator yu bees yi soo koy méngale ak njëgu jeexal memory bi. Kadre yu melni PyTorch (torch.utils.checkpoint) dañuy laxas benn modle suko defee ñu denc limu génne ci kanam waaye li ci biir dañu koy xaymaat ci ginaaw. Tann plasement checkpoint lu am solo la: ab diggante bu tolloo ci sqrt(N) segments dafay wàññi memory bi yépp ci di yokk benn pass forward bu gëna mag ci calcul bi.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu saytu gradient
Checkpointing gradient leegi dafay nekk standard ci tàggat model yu mag yi te dafay gëna otomatise, bibliotek yi di tànnee barabi checkpoint yi gëna baax ci yaw. Dafay boole ci anam wu natureel ak FSDP, jaxaso bu jaar yoon, ak dechargement ngir push model yu gëna kawe. Xaarandi 'tanneef' checkpointing biy xaymawaat liggéey yu yomb yi di denc yu seer yi (lu melni matris yiy bàyyi xel) ci cache, boole ci jegewaale yu compilatër bi di dawal ci jumtukaay yu melni PyTorch's torch.compile biy jël dogal ci saasi li ñuy denc ak xaymawaat ngir am balance memory bu gëna gaaw.
Doxal ci àdduna dëgg
Taggat ab transformatër bu xóot ak dayo lote bu gëna mag ci benn GPU ci sànni ak xaymawaat aktivaasioŋu couche yi.
Modèlu gis-gis bu jaar yoon ci kaw nataal yu am dayo bu kawe, fu kàrtu aktivasioŋ yi di fees ci mémoire GPU bi.
Transformatëri kanam yuy laxasu ñuy may gradient_checkpointing=Dëgg ngir méngoo ak xeetu paramet yu am miliyaar ci diiru ajustement bu baax.
boole checkpointing ak FSDP suko defee parametre yi ak activation yi nekk ñu tuuti, loolu mooy tax ñu mëna tàggat modeli làkk yu yaatu lool.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Gradient Checkpointing quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Dajale degrade
Laaj yi ñuy faral di laaj
What is Gradient Checkpointing?
Checkpointing gradient (ñu koy woowe itam checkpointing activation) ab pexe la buy sakkanal mémoire biy sànni activation yu bari yi ci digg bi ci diir bi ñuy jaar ci kanam, ba noppi di leen xaymaat ci diir bi ñuy tasaare ci ginaaw. Daf lay may nga tàggat reso yu gëna xóot te gëna mag ci jënd ay ordinatër yu gëna bari ngir wàññi memory bi.
Lan mooy njëkk jënd ak jaay gradient checkpointing ngir sakkanal mémoire?
Gradient checkpointing dafay xaymawaat aktivaasioŋ yiñ sànnee ci diiru paas bi ci ginaaw, di dugal xaalis bu gëna bari ngir am memory bu wàññeeku bu baax.
Lan moo waral ñuy denc aktivaasioŋ yi ci diiru jàll bi?
Backprop dafay xayma gradient yi ci diggante aktivasioŋ yi ci paas bi ci kanam, kon dañu wara am fileek xayma wuñu leen.
Ci gàttal, naka la mémoire bi di doxee sudee dañu def ay checkpoint ci sqrt(N) bu nekk ci biir reso bu N-layer?
Diggante poñ yi ñuy saytu ci bepp racine-carré-de-N layers dafay wàññi memory activation biñ denc daale ko ci rang N wàcci ba ci rang sqrt(N).
Lu tollu ci ñaata xayma yu gëna bari la checkpointing gradient biñ def bu baax di yokk?
Ak plasement bu baax ci checkpoint, li ngay fay ci kaw mooy benn yoon kese ngay yokk ci kanam, lu bari ci 20-30% ci yeexal.
Ci PyTorch, ban njariñ lañuy faral di jëfandikoo ngir saytu gradient ci benn modle?
torch.utils.checkpoint dafay laxas benn modle suko defee ñu mëna xaymawaat ay aktivaasioŋ yi ci biir ci diir bi ñuy delloo ginaaw, duñu ko denc.