Aktiwisaasioŋ xaymawaat kompromis
Recomputation biy tàmbali (gradient wala checkpointing biy tàmbali) dafay sakkanal mémoire GPU bi ci diiru tàggat yaram, ci sànni diggante yi ci paas bi ci kanam, ba noppi di leen xaymawaat ci paas bi ci ginaaw.
Résumé
It trades extra compute for the ability to train larger models or longer sequences on the same hardware.
Plongeur bu xóot
Backpropagation mingi soxla aktivaasioŋ forward-pass ngir xayma gradient yi, kon ci default bépp couche génne dañu koy denc - njëgu mémoire bu rëy buy màgg ak dayo model bi, dayo batch bi, ak guddaayi sequence bi. Recomputation biy aktive dafay denc yenn tensor 'checkpoint' (dafay faral di nekk ay peggu layer) ba noppi sànni leneen li. Ci diiru paas bi ci ginaaw, dafay defaraat xayma bi ci kanam diggante poñ yi ngir defaraat activasioŋ yiñ sànni ci laaj. Resultaa bi gëna mag mooy suñu defee ay checkpoint ci bépp sqrt (N) layer, mémoire bi dafay wàcci ba O (sqrt (N)) ci noonu lañuy yokk lu tollu ci benn pass forward (~ 33% gëna calcul). Variante yuñ tànnee recompute ops yu yomb waaye-memoire-diis (lu melni bàyyi xel wala dropout) ci cache yu seer yi, jot li gëna bari ci memory bi ñuy sakkanal ngir gëna néew recompute.
Gis-gis xarala
Kompromis bi gëna am solo mooy memory ak FLOPs. Recomputation bu mat sëkk dafay yokk benn pass ci kanam ci jéego bu nekk (~30-40% gëna ndànk) waaye mën na dagg mémoire biy aktive ci xeetu magnitude. Demaas bu am xel bi mooy tànneef xool: ràññee ops yu am memory bu yaatu waaye xayma-yomb (softmax, layernorm, GELU, poñ yi) ak xaymawaat ñoom rek, boole ci denc njariñu GEMM yu seer yi ci cache - wàññi xayma bu yàqu.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu njaay ci xaymawaat
Recomputation dafay gëna otomatise ak tànneef. Leegi kaadar yi dañuy profile mémoire op bu nekk ak njëgu FLOP ngir tànn poñ yi gëna baax, ba noppi boole xaymawaat ak jëfandikoo CPU/NVMe ak pexe parallelism. Bi guddaayi contexte ak dayo model di gëna màgg, xaarandil politik yu compilatër bi di dawal (ci PyTorch, JAX / XLA) yuy tànn ci saasi dogal yi ci operaasioŋ bu nekk, boole ci lëkkaloo bu gëna dëgër ci xayma ak jokkoo suko defee FLOP yu bari yi dañuy nëbbu.
Doxal ci àdduna dëgg
Taggat ab transformatër bu rëy budul mëna ànd ci saytu bépp blok layer
Jëfandikoo torch.utils.checkpoint bu PyTorch ngir ëmb ay bloku trafo ak dagg mémoire biy tàmbali
Tanneef xaymawaat bàyyi xel/softmax ci Megatron-LM ngir denc mémoire ak yeexal bu néew
Aktiwise guddaayi sekans yu gëna gudd ci budget GPU bu takku ci xaymawaat aktivaasioŋ yi ci barabu denc leen
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Activation Recomputation Tradeoffs quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
SmoothQuant ak jëfandikoo limu
Laaj yi ñuy faral di laaj
What is Activation Recomputation Tradeoffs?
Recomputation biy tàmbali (gradient wala checkpointing biy tàmbali) dafay sakkanal mémoire GPU bi ci diiru tàggat yaram, ci sànni diggante yi ci paas bi ci kanam, ba noppi di leen xaymawaat ci paas bi ci ginaaw. Dafay wecci yeneen ordinatër ngir mëna tàggat model yu gëna mag wala sequence yu gëna gudd ci benn hardware bi.
Lan la recomputation biy aktive ngir sakkanal mémoire bi?
Recomputation dafay sànni activation yiñ denc ba noppi defaraat leen ci backward pass bi, di jëfandikoo calcul bu gëna bari ngir wàññi jëfandikoo mémoire bi.
Lan moo waral ñuy denc aktivaasioŋ forward-pass yi ci anam wu jaar yoon?
Pass bi ci ginaaw dafay jëfandikoo activation yi ci kanam ngir xayma gradient yi, kon ci default dañu leen di denc ci mémoire bi ba pass bi ci ginaaw bi di dox.
Lu tollu ci ñaata calcul yu bees la recomputation activation bu mat bi di yokk?
Recomputation bu mat sëkk dafay doxalaat calcul bi ci kanam ci diiru paas bi ci ginaaw, yokk ci lu tollu ci benn paas ci kanam bu gëna bari - ci rang bu 30-40% ci calcul bi gëna bari.
Lan mooy xalaat bi ci ginaaw tànneef (bu matwul)
Recomputation bu tanneef dafay xool ops yiy jëfandikoo memory bu bari waaye calcul bu néew (lu melni softmax wala layernorm), ci noonu lañuy cache ay resultaa GEMM yu seer ngir wàññi FLOP yu yàqu.
Ban pexem mottali lañuy faral di boole ak xaymawaat ngir sakkanal memory bu gëna bari?
Deffloading biy tàmbali dafay toxal yenn aktivasioŋ yi ci dencukaay CPU/NVMe, te dañu koy faral di boole ak recomputation ak parallelism ngir gëna sakkanal mémoire bi.