Ci bi mu melGis bi ci topp
Estimating GPU Memory Needed for LLMs
Xarala
GUIDE teknik
Memoire bu yaatu
Memoire Bandwidth bu kawe (HBM) mooy mémoire buñ def ci wetu GPU biy joxe done yi gëna gaaw ci RAM yi.
Déglul tegtal biiIA-nettali · ~2 min
Ci xët wii2 simili jàng
Résumé
It is what keeps AI accelerators fed, preventing the powerful compute cores from sitting idle while they wait for model weights and data.
Plongeur bu xóot
HBM saafara na jafe-jafe bu yomb: puce IA yu bees yi mën nañu def ay trillion ci operation ci segond bu nekk, waaye sudee done yi yegsi nañu ci lu gaaw. Memoire GDDR buñ miin dafay boole ci bus bu sew, waaye HBM dafay boole DRAM yu bari ci anam wu taxaw ba noppi boole leen ci ay junni fiil yu ndaw yu taxaw yu ñuy woowe through-silicon vias (TSVs). Stack yooyu ñu ngi toog ci kaw interposer silicon millimetre ci GPU bi, di joxe yoonu done bu yaatu lool, xalaat ay junni bit ci benn yoon ci barabu téemeeri. Lépp soo ko boolee mu am yaatuwaayu band bi ñuy natt ci terabyte ci segond bu nekk. Ay jamono joge ci HBM2 dem ba ci HBM2e, HBM3, ak HBM3e, bu nekk di yokk kàttan ak gaawaay. Ci modelu làkk yu mag yi, te dañu wara streaming seeni poid saa yu nekk, kàttanu HBM ak yaatuwaayu bandwidth dañu gëna am solo ci ordinatër bu ñor.
Doxal ci àdduna dëgg
Teg fukki wala téemeeri gigabytes ci diisaay ngir modelu làkk bu mag bu jege GPU bi suko defee ñu mëna streaming ci bépp jéego bu jëm ci inference.
Fexe ba NVIDIA H100 ak H200 GPU yi mëna yegg ci terabyte yu bari ci segond bu nekk ci yaatuwaayu mémoire ngir tàggat.
Xootal clusters tàggat IA fu GPU yu bari ku nekk di wéeru ci HBM ngir moytu taxaw ci diggante liggéey matrix yi.
Jàppale nataal yu am resolusioŋ bu kawe ak modeli wideo yu wara toxal tensor yu mag yuy tàmbali ak genne ci mémoire bi ci lu gaaw.
Ëlëgu mémoire bu bari bandwidth
Yaatu bandwidth mémoire mooy gëna gàllankoor IA, moo tax HBM mingi awaase bu baax. HBM3e mingi yóbbu ci gaawaayu flagship, ak HBM4 ci horizon bi dige interface yu gëna yaatu, stack yu gëna njool, ak kàttan gu gëna bari ci koli bu nekk. Xaarandil jëmmal bu gëna jege diggante mémoire ak logique, amaana base dies ak defar-memoire bu jege, boole ci joŋante bu metti diggante furnisër yu melni SK hynix, Samsung, ak Micron. Lu model yi di màgg, gëna am done yu gëna jege ordinatër, gëna gaaw ak energie bu néew, mooy nekk lu am solo ci yokkuteg hardware IA.
Teg ko ci jëf
Lu tax mu am solo
- Njëgg ak budget. Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
- dogal yu gëna leer. Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
- Xool kalite. Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Moytul
- Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
- Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
- Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Jéego yi ci topp
- Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
- Benchmark ci biir sargal ak done yu dëggu.
- Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
- Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Bulletin bu amul fayda
Jàngal xibaar AI bu bes
Ñatti jaar-jaari IA yuñ firndeel bis bu nekk ci ayu-bis bi, ñu bind leen ci Àngle bu leer. Gratis ba fàww, amul yëgle.
Benn email bis bu nekk ci ayu-bis bi. Benn klik nga dindi abonemaa bi. Dunu musa jaay wala séddoo sa adres.
Nattal sa bopp
Take the High Bandwidth Memory quiz
Feedback ci saasi ci bépp tontu, ak sertifikaa buñ mëna séddoo ak ID buñ mëna saytu soo jàllee ci njàng mi.
Jàppale njàngum IA te doo fay. AI Understanding mbootaay la buy def te Yàlla tax — amul yëgle, amul paywall, musul am. Defal ab sarax
Laaj yi ñuy faral di laaj
What is High Bandwidth Memory?
Memoire Bandwidth bu kawe (HBM) mooy mémoire buñ def ci wetu GPU biy joxe done yi gëna gaaw ci RAM yi. Mooy luy dundal gaawaayu IA yi, di tere core ordinatër yu am doole yi toog rek di xaar diisaayu model yi ak done yi.
Lan mooy jafe-jafe bi gëna mag bi Memoire Bandwidth bu kawe di saafara ci gaawaayu IA yi?
Puce IA yi mën nañu def ay trillioŋu liggéey ci segond bu nekk sudee done yi gaaw nañu yegsi; HBM dafay joxe bandwidth boobu suko defee core yi duñu xiif.
Lan moo wuutale HBM ak mémoire GDDR bi?
HBM dafay boole DRAM yi benn ci kaw beneen bi ba noppi boole leen ak ay junni kable yu taxaw (TSVs), sos yoonu done bu yaatu lool.
Lan la HBM gëna yéem ngir mëna am bandwidth bu rëy?
Du gëna gaaw, HBM dafay wane interface bu yaatu lool (1024 bits ci stack bu nekk ak ci kaw) ba byte yu bari di toxu benn yoon.
Lan moo waral ñuy def ay pilu HBM ci kaw ab interposer silisium ci wetu GPU bi?
Fiil yu gàtt yi ci kaw interposer buñ bokk dañuy wàññi energie bi ñuy soxla ci bit bu nekk buñ toxal ba noppi wàññi latency bi am ci diggante mémoire bi ak ordinatër bi.
Ci wàllu làkk yu mag, lu tax HBM mëna am solo ni ordinatër bu ñor?
LLM inference dafay wéy di génne matris yu yaatu, moo tax kàttan mémoire ak yaatuwaayu band dañuy faral di nekk facteur limitant bi moo gën ñuy xayma produit yi.
Weyal di jàng
Gid yu jëm ci loolu
Tann nañu yeneen njiit ngir topic bii