Njikwa ebe nchekwa GPU na nkewa
Kedu ka usoro AI na-ekenye, jigharịa, ma nwetaghachi ebe nchekwa nwere oke na GPU, yana ihe kpatara oghere efu (nkewa) nwere ike ibute njehie na-enweghị ncheta ọbụlagodi mgbe ọtụtụ ebe nchekwa dị na teknụzụ.
Nchịkọta
Understanding it is key to fitting big models and avoiding mysterious crashes.
Ime miri emi
Ebe nchekwa GPU bụ nke edobere ma dị oke ọnụ ahịa: kaadị nwere ike ịnwe mkpokọta 24, 80, ma ọ bụ 192 GB, na-ekekọrịta site na nha ihe atụ, mmemme, gradients, steeti ndị na-ebuli elu, yana nchekwa nwa oge. Ịkpọ onye ọkwọ ụgbọ ala ka ọ kenye ebe nchekwa na ọrụ ọ bụla ga-adị ngwa ngwa, ya mere, usoro dị ka PyTorch na-eji ihe nkwụnye ego na-ejide nnukwu ihe mgbochi n'ihu ma na-enye obere mpempe akwụkwọ, wee na-edobe iberibe n'ime ọdọ mmiri maka ijikwa ya. Ihe nwude ahụ bụ nkewa: ka a na-ekenye tenors nke nha dịgasị iche iche ma tọhapụ ya, ohere ahụ efu na-agbaji n'ime mpekere gbasasịrị. Ị nwere ike ịnweta 5 GB n'efu na mkpokọta ma ị gaghị ekenye 2 GB tensor na-aga n'ihu n'ihi na ọ nweghị oghere zuru oke. Nke a bụ ya mere ọzụzụ nwere ike ịdaba na njehie na-enweghị ncheta n'agbanyeghị na ọ dị ka isi ụlọ.
Nghọta nka nka
PyTorch's CUDA caching allocator na-ekewa ebe nchekwa n'ime iyi nke ngọngọ ma na-ejigharị ihe mgbochi ndị nwere onwe ha dabara na nha achọrọ, na-ezere oku cudaMalloc/cuda efu. Nkewa na-ebilite mgbe enweghị ike ijikọ ngọngọ gbawara agbawa. Ngwa dị ka torch.cuda.empty_cache, nhọrọ PYTORCH_CUDA_ALLOC_CONF expandable_segments, na ebe nchekwa snapshots na-enyere aka. Ụzọ ọhụrụ dị ọhụrụ na-agbaziri echiche mebere-ncheta, na-esepụta ibe anụ ahụ na-abụghị nke na-aga n'ihu n'ime usoro mebere nke ọma ka nnukwu arịrịọ wee gaa nke ọma n'agbanyeghị nkewa.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke njikwa ebe nchekwa GPU na nkewa
Njikwa ebe nchekwa na-aghọwanye nkọ ma na-edobekwu ibe ya, sitere na sistemụ arụmọrụ. Usoro dị ka ndị na-eke ụdị memory-memory na nlebara anya ihu (nke a na-eji ejikwa cache KV n'oge ntinye) na-ebelata mkpofu na nkewa nke ukwuu. Na-atụ anya ka usoro ga-adabara ka ọ bụrụ nke a na-agbasawanye, ndị na-emebi emebi, visibiliti ka mma site na profaịlụ arụnyere arụnyere, yana njikọ siri ike na nbudata na nkwughachi ka sistemụ ahụ na-agbakọ GPU, CPU, na ebe nchekwa diski na-akpaghị aka iji mee ka ojiji dị elu ma daa ụkọ.
Mmejuputa n'ezie n'ụwa
Ọsọ ọzụzụ nke na-ada na 'CUDA enweghị ebe nchekwa' n'agbanyeghị ebe nchekwa echekwara na-egosi oghere efu, edoziri site na ịtọ PYTORCH_CUDA_ALLOC_CONF iji mee ka akụkụ nwere ike gbasaa.
Iji torch.cuda.memory_summary ma ọ bụ foto ebe nchekwa iji chọpụta nke tenors na nkewa na-eri 80 GB nke GPU.
vLLM's Paged Ntị nlebara anya ijikwa oghere KV nlebara anya na ibe ndị nwere oke iji jeere ọtụtụ arịrịọ nkata n'otu oge na-emefusịghị ebe nchekwa.
Iwedata nha batch ma ọ bụ na-eme ka nlele gradient belata ebe nchekwa ọrụ wee zere ndabichi na-esiteghị na ncheta.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the GPU Memory Management and Fragmentation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Usoro nhazi GPU na nhazi ụyọkọ
Ajụjụ a na-ajụkarị
What is GPU Memory Management and Fragmentation?
Kedu ka usoro AI na-ekenye, jigharịa, ma nwetaghachi ebe nchekwa nwere oke na GPU, yana ihe kpatara oghere efu (nkewa) nwere ike ibute njehie na-enweghị ncheta ọbụlagodi mgbe ọtụtụ ebe nchekwa dị na teknụzụ. Ịghọta ya bụ isi ihe na-adaba nnukwu ụdị na izere mkpọka dị omimi.
Gịnị bụ GPU ebe nchekwa fragmentation?
Nkewa pụtara na mkpokọta ebe nchekwa efu ezuola, mana a na-agbaji ya n'ime iberibe, yabụ na ọ nweghị oghere zuru oke maka nnukwu tensor.
Kedu ihe kpatara usoro dị ka PyTorch ji eji ihe nchekwa nchekwa nchekwa?
Ịkpọ cudaMalloc/cudaFree maka op ọ bụla na-adị ngwa ngwa, ya mere onye na-ahụ maka caching na-ejide nnukwu ihe mgbochi ma na-ejigharị ndị a tọhapụrụ n'ọdọ mmiri.
Ị na-ahụ 5 GB n'efu mana enweghị ike ịkenye tensor 2 GB. Gịnị nwere ike ịkpata?
Ihe mgbaàmà a kpochapụrụ nke nkewa na-apụta mgbe ebe nchekwa efu dị mana ọ bụghị dị ka otu mkpọ na-aga n'ihu buru ibu maka arịrịọ ahụ.
Kedu ntọala PyTorch na-enyere aka belata nkewa site n'ikwe ka akụkụ ebe nchekwa na-eto n'ụzọ na-agbanwe agbanwe?
Ịtọ ntọala PYTORCH_CUDA_ALLOC_CONF iji mee ka akụkụ nke expandable na-eme ka onye na-ekenye ihe tolite akụkụ ma belata ọdịda metụtara nkewa.
Kedu ihe vLLM's PagedAttention jikwaa ibelata ihe mkpofu ebe nchekwa n'oge ntinye?
PagedNttention na-echekwa cache KV na ibe ndị nwere oke dị ka OS mebere ebe nchekwa, belata nkewa na ijere ọtụtụ arịrịọ nke ọma.