Doxal mémoire GPU ak xaaj
Noo ngi xaajalee, jëfandikoowaat, ak jëlaat memory bu néew bi ci GPU, ak lu tax ay bërëb yu des (fragmentation) mën na jur njuumte ci memory bi doonte memory bu bari des ci xarala.
Résumé
Understanding it is key to fitting big models and avoiding mysterious crashes.
Plongeur bu xóot
Memoire GPU dafa yàgg te am solo: benn kart mën na am 24, 80, wala 192 GB ci lëmm, ñu bokk ko ci diisaayu model, aktivasioŋ, gradient, staadu optimisatër, ak tampon yu yàgg. Woo dawalkat bi ngir xaaj mémoire ci bépp liggéey dina yeex, kon kaadar yu melni PyTorch dañuy jëfandikoo ab xaaj cache buy jàpp blok yu rëy ci kanam ba noppi joxe ay piyees yu ndaw, ba noppi denc piyees yi ñu bàyyi ci piscine ngir jëfandikoowaat. Liy jàpp mooy fragmentation: ginaaw bi ñu xaajalee ay tensor yu bari te wuute, ba noppi ñu bàyyi leen, bayaal bi amul dara dafay xaajaloo nekk ay pàcc yu tasaaroo. Mën nga am 5 GB ci lëmm waaye doo mëna joxe 2 GB tensor bu laal ndax amul benn gap bu doy. Lii moo waral tàggat yaram mën na tass ak njuumte yu génn ci memory doonte daa melni am bayaal ci bopp.
Gis-gis xarala
PyTorch's CUDA cache allocator dafay xaaj mémoire bi ci ay bloc yu bari ba noppi jëfandikoowaat bloc yiñ bàyyi ñu méngoo ak dayo yiñ laaj, moytu woote cudaMalloc/cudaFree yu seer. Fragmentation dafay am sudee blok yuñ xaaj mënu ñu boolewaat. Jumtukaay yu melni torch.cuda.cache_bu amul dara, tànneef PYTORCH_CUDA_ALLOC_CONF, ak nataali mémoire yi dañuy jàppale. Xeetu jëfandikoo yu bees yi dañu leble xalaati mémoire virtuel, di boole xët physique yi nekkul ci benn rang virtuel bu jege, suko defee laaj yu bari mëna dem doonte dañu xaajaloo.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu GPU jëfandikoo mémoire ak xaaj
Doxalal mémoire mingi gëna xarañ, gëna am xët, te dafa lalu ci sistem operaasioŋ yi. Pexe yu melni allocateurs yu nuroo ak mémoire virtuel ak paged attention (ñu koy jëfandikoo ngir yoriinu cache KV ci jamonoy inference) dañuy wàññi bu baax waste ak fragmentation. Xaarandi kaadar yi ñu mëna yaatal, defragmenter allocators, gëna gis ci profiler yiñ tabax, ak lëkkaloo bu gëna dëgër ak dechargement ak recomputation suko defee sistem bi di juggling GPU, CPU, ak disk memory ci saasi ngir wéy di jëfandikoo lu bari te daanu.
Doxal ci àdduna dëgg
Taggat yaram buy tas ak 'CUDA bi génn ci mémoire' doonte mémoire biñ denc dafay wane bayaal bu amul dara, ñu defar ko ci def PYTORCH_CUDA_ALLOC_CONF ngir mëna yokk segment yi.
Jëfandikool torch.cuda.memory_summary wala ab nataalu mémoire ngir xam ban tensor ak xaaj mooy lekk 80 GB ci GPU bi.
vLLM's PagedAttention dafay yor cache KV ci xët yu am dayo bu takku ngir mëna jàppale chat yu bari te baña yàq memory.
Wàññil dayo lots bi wala nga may poñ yiñy saytu ci gradient ngir dagg mémoire biy aktive ak moytu fragmentation biy bawoo ci paj mu génn ci mémoire bi.
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the GPU Memory Management and Fragmentation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
GPU ak orkestraasioŋ cluster
Laaj yi ñuy faral di laaj
What is GPU Memory Management and Fragmentation?
Noo ngi xaajalee, jëfandikoowaat, ak jëlaat memory bu néew bi ci GPU, ak lu tax ay bërëb yu des (fragmentation) mën na jur njuumte ci memory bi doonte memory bu bari des ci xarala. Xam ko mooy caabi ngir mëna ànd ak model yu mag yi ak moytu accident yu yéeme.
Luy xaajalug mémoire GPU?
Fragmentation dafay tekki ni mémoire bi doyna, waaye dafay xaajaloo ay piyees, kon benn gap bu doyul ngir tensor bu rëy.
Lan moo waral kaadar yu melni PyTorch di jëfandikoo ab tànneefu mémoire cache?
Woo cudaMalloc / cudaFree ngir bépp op dafa yeex, moo tax allokatëru cache dafay jàpp blok yu rëy ba noppi jëfandikoowaat ñi ñu génne ci piscine bi.
Danga gis 5 GB te doo fay waaye mënoo joxe 2 GB tensor. Lu mëna waral loolu?
Bii màndarga bu yàgg bu fragmentation dafay am su amee memory bu amul benn coono waaye du benn chunk bu doy ngir laaj bi.
Ban jekkal PyTorch mooy wàññi xaajaloo bi ci may segmenti memory yi ñu màgg ci anam wu yomb?
Regler PYTORCH_CUDA_ALLOC_CONF ngir may wàll yiñ mëna yaatal dafay tax xaajkat bi màgg wàll yi ba noppi wàññi jafe-jafe yi jëm ci xaaj.
Lan la PagedAttention bu vLLM mëna def ngir wàññi yàqu-yàqu ci mémoire bi ñuy jël dogal?
PagedAttention dafay denc cache KV ci xët yu am yaatuwaayu OS lu melni mémoire virtuel OS, dagg xaaj-xaaj bi ak def ay laaj yu bari ci anam wu jaar yoon.