Gudanar da Ƙwaƙwalwar Ƙwaƙwalwar GPU da Rarraba
Yadda tsarin AI ke keɓancewa, sake amfani da shi, da dawo da ƙayyadaddun ƙwaƙwalwar ajiya akan GPU, kuma me yasa raguwar raguwa (raguwa) na iya haifar da kurakuran ƙwaƙwalwar ajiya ko da lokacin da yawan ƙwaƙwalwar fasaha ta rage.
Dubawa
Understanding it is key to fitting big models and avoiding mysterious crashes.
Zurfafa nutsewa
Ƙwaƙwalwar GPU tana da ƙayyadaddun ƙayyadaddun ƙayyadaddun ƙwaƙwalwar ajiya da daraja: katin zai iya samun jimlar 24, 80, ko 192 GB, wanda aka raba ta ma'aunin ƙira, kunnawa, gradients, jihohin ingantawa, da maɓalli na wucin gadi. Kiran direba don keɓance ƙwaƙwalwar ajiya akan kowane aiki zai kasance a hankali, don haka tsarin tsarin kamar PyTorch suna amfani da allocator na caching wanda ke ɗaukar manyan tubalan gaba da fitar da ƙananan yanki, sannan adana ɓangarorin da aka saki a cikin tafkin don sake amfani da su. Kama yana rarrabuwar kawuna: yayin da aka keɓe tenors na masu girma dabam dabam da kuma 'yantar da su, sararin sararin samaniya ya karye zuwa ɓarke zuwa warwatse. Kuna iya samun 5 GB kyauta gabaɗaya duk da haka kuna kasa ware madaidaicin 2 GB tensor saboda babu tazari ɗaya da ya isa. Wannan shine dalilin da ya sa horarwa na iya yin karo tare da kurakuran da ba a iya tunawa ba duk da da alama akwai dakin kai.
Fahimtar Fasaha
PyTorch's CUDA caching allocator yana raba ƙwaƙwalwar ajiya zuwa rafukan tubalan kuma yana sake amfani da tubalan da aka saki waɗanda suka dace da girman da ake buƙata, guje wa kiraye-kirayen cudaMalloc/cuda masu tsada. Ragewa yana tasowa lokacin da ba za a iya sake haɗa tubalan ba. Kayayyakin aiki kamar torch.cuda.empty_cache, PYTORCH_CUDA_ALLOC_CONF zažužžukan fadada_segments, da hotunan ƙwaƙwalwar ajiya suna taimakawa. Sabbin hanyoyi suna aro ra'ayoyin memori mai kama-da-wane, yin taswirar shafukan zahiri marasa ci gaba zuwa cikin kewayon kama-da-wane don haka manyan buƙatun sun yi nasara duk da rarrabuwa.
Dabarun Tasiri
Kudin da kasafin kuɗi
Hukunce-hukuncen gine-gine suna haifar da aiki da tsadar aiki na shekaru.
Shawarwari masu haske
Ilimin fasaha yana taimaka wa ƙungiyoyi su zaɓi tari mai kyau, ba kawai sabon abu ba.
Kula da inganci
Zaɓuɓɓukan injiniya mafi kyau suna rage abin dogaro a cikin samarwa.
Makomar Gudanar da Ƙwaƙwalwar Ƙwaƙwalwar GPU da Rarraba
Gudanar da ƙwaƙwalwar ajiya yana ƙara wayo kuma yana da ƙarin shafi, wahayi daga tsarin aiki. Dabaru kamar masu rarraba salon-memory mai kama-da-wane da kulawar shafi (wanda aka yi amfani da su don sarrafa ma'ajin KV yayin tantancewa) suna rage sharar gida da rarrabuwa sosai. Yi tsammanin ginshiƙai zuwa tsoho don faɗaɗawa, masu ɓarna masu rarrabawa, mafi kyawun gani ta hanyar ginannun bayanan martaba, da ƙarin haɗin kai tare da saukewa da sake lissafin don haka tsarin yana jujjuya GPU, CPU, da ƙwaƙwalwar faifai ta atomatik don ci gaba da amfani da girma kuma ba kasafai ba.
Aiwatar da Gaskiyar Duniya
Gudun horon da ya yi karo da 'CUDA ba ta da ƙwaƙwalwar ajiya' duk da tanadin ƙwaƙwalwar ajiya yana nuna sarari kyauta, an daidaita shi ta hanyar saita PYTORCH_CUDA_ALLOC_CONF don kunna sassan da za a iya faɗaɗawa.
Amfani da torch.cuda.memory_summary ko hoton ƙwaƙwalwar ajiya don tantance waɗanne tenors da rarrabuwa ke cin GPU's 80 GB.
vLLM's Paged Hankali yana sarrafa ma'ajiyar kulawar KV a cikin ƙayyadaddun shafuka masu girma don ba da buƙatun taɗi na lokaci ɗaya ba tare da ɓata ƙwaƙwalwar ajiya ba.
Rage girman batch ko ba da damar duban gradient don yanke ƙwaƙwalwar kunnawa da guje wa ɓarna-kore gazawar ƙwaƙwalwar ajiya.
Hatsari & Tsare-tsare
Haɓaka ma'auni ɗaya na iya ɓoye manyan raunin tsarin.
Sau da yawa ana raina kayan more rayuwa da kuma kuɗin kulawa.
Tsaro da gibin lura na iya girma yayin da tsarin ke ƙara haɓaka.
Taswirar Hanya
Ƙayyade latency, inganci, da maƙasudin farashi kafin aiwatarwa.
Alamar ma'auni a ƙarƙashin ainihin kaya da yanayin bayanai.
Kula da kayan aiki don kurakurai, ɗigo, da tasirin mai amfani.
Shirya bijirowa da hanyoyin mayar da martani kafin sikeli.
Ci gaba da Bincike
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the GPU Memory Management and Fragmentation quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Jagora na gaba
Jadawalin GPU da Ƙungiyoyin Ƙira
Tambayoyin da ake yawan yi
What is GPU Memory Management and Fragmentation?
Yadda tsarin AI ke keɓancewa, sake amfani da shi, da dawo da ƙayyadaddun ƙwaƙwalwar ajiya akan GPU, kuma me yasa raguwar raguwa (raguwa) na iya haifar da kurakuran ƙwaƙwalwar ajiya ko da lokacin da yawan ƙwaƙwalwar fasaha ta rage. Fahimtar shi shine mabuɗin don dacewa da manyan samfura da guje wa ɓarna masu ban mamaki.
Menene rarrabuwar ƙwaƙwalwar GPU?
Ragewa yana nufin jimlar ƙwaƙwalwar ajiyar kyauta ta isa, amma an karye shi guda, don haka babu tazari ɗaya da ya isa babban tensor.
Me yasa tsarin tsarin kamar PyTorch ke amfani da mai rarraba ƙwaƙwalwar ajiya?
Kira cudaMalloc/cudaFree ga kowane op yana jinkirin, don haka mai rarraba caching yana ɗaukar manyan tubalan kuma yana sake amfani da waɗanda aka 'yantar daga tafkin.
Kuna ganin 5 GB kyauta amma ba za ku iya rarraba tensor 2 GB ba. Menene yuwuwar sanadin hakan?
Wannan al'adar alama ta rarrabuwa tana faruwa lokacin da ƙwaƙwalwar ajiyar kyauta ta wanzu amma ba azaman guda ɗaya mai girma isa ga buƙatar ba.
Wanne saitin PyTorch yana taimakawa rage rarrabuwa ta hanyar barin sassan ƙwaƙwalwar ajiya suyi girma cikin sassauƙa?
Saita PYTORCH_CUDA_ALLOC_CONF don ba da damar fadada_segments yana bawa mai rabo damar girma sassa kuma yana rage gazawar da ke da alaƙa.
Menene vLLM's PagedAttention ya sarrafa don rage sharar ƙwaƙwalwar ajiya yayin ƙaddamarwa?
PagedAttention yana adana ma'ajin KV a cikin ƙayyadaddun shafuka kamar ƙwaƙwalwar ajiya na OS, yanke rarrabuwa da ba da buƙatu da yawa yadda ya kamata.