Batching bu wéy
Batching bu wéy bi pexem serwiis la buy yokk wala dindi ay laaj ci batch buy daw token-by-token, ci plaasu xaar batch bu mat sëkk jeex.
Résumé
It keeps the GPU constantly busy and sharply increases how many users an AI model can serve at once.
Plongeur bu xóot
GPU yi ñoo gëna gaaw suñu boolee ay laaj yu bari ci benn batch. Approche naïf bi, batching static, dafay boole ay laajte yu takku, def leen ñépp bañu jeex, ba noppi tàmbali batch bi ci topp. Jafe-jafe bi: génnug modelu làkk yi dañu wuute lool ci guddaay, moo tax laaj yu gàtt yi dañuy teela jeex, seeni barab dañuy toog ba noppi batch bi di xaar ki gëna gudd, di yàq cycle GPU yi ak yeexal ñëw yu bees yi. Batching bu wéy (ñu koy woowe itam batching ci naaw wala ci niveau iteration, siiwal ko ci këyit Orca te ñu jëfandikoo ko ci vLLM, TensorRT-LLM, ak TGI) dafay dox ci granularite bu benn jéego dekodaas. Ginaaw ñu defaree token bu nekk, sequences yu jeex yi dañuy génn ci lot bi, ba noppi ñuy dugal ci saasi laajte yu bees yi. Loolu dafay tax batch bi fees ba noppi GPU bi fees, di yokk produit bi lu bari yoon ci latency bu néew ngir jëfandikukat yiy xaar.
Gis-gis xarala
Coppite bu am solo bi mooy joge ci batching laajte yépp dem batching iterations benn-benn. Bépp jéego decode, scheduler bi dafay tabax ensemble biy dox: dafay daw benn pass forward ci kaw bépp sequence in-flight, génne benn token bu nekk, dàq bépp token bu jeexal sequence bi wala àppu guddaay bi, ba noppi nangu laaj yi ñu raŋ ngir feesal barab yiñ bàyyi. Soo boole lii ak mémoire KV bu yomb bu PagedAttention, dafay tax dugal ak dindi ay toppalante ci diggu naaw gi yomb, ndax cache bu toppalante bu nekk mingi dëkk ci ay blok yu moom seen bopp.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu batching bu wéy
batching wéy leegi nekk na standard ci liggéey LLM. Liggéey bi ëlëg dafay setal scheduler bi: tàqale fasu prefill bu diis bi ak fasu decode bu woyof bi (disaggregation), prefill chunked ngir moytu decodage, politik yu njëkk ak yoon ngir liggéey yu bari, ak couplage bu gëna dëgër ak decodage speculatif, kon jéego yu bari lañu. Luñu bëgga mooy gëna bari ay token ci segond bu nekk ci GPU bu nekk, boole ci tëye latency tontu bu benn-benn bi, te mën nañu ko seetlu.
Doxal ci àdduna dëgg
API chat buy nangu mesaas jëfandikukat yu bees yi ci lote biy dawal ci saasi ci barabu leen raŋ ngir lote bi ci topp
Dëddu ab tontu bu gàtt bu mat ci diggu lote bi ak delloosi xar kanamam suko defee GPU bi baña musa nekk ci liggéey di xaar ab jamono bu yàgg
boole batching bu wéy ak vLLM's PagedAttention ngir dugal ak dindi toppalante ci anam wu yomb ci jéego bu nekk ci dekode
ab sarwis buy yeggali kode buy tëye ay token yu bari ci segond bu nekk ci suufu trafik bu bari te wuute guddaay bi di tëye lote bi fees
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continuous Batching quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Kubernetes ngir ML
Laaj yi ñuy faral di laaj
What is Continuous Batching?
Batching bu wéy bi pexem serwiis la buy yokk wala dindi ay laaj ci batch buy daw token-by-token, ci plaasu xaar batch bu mat sëkk jeex. Dafay tax GPU bi di wéy di liggéey, ba noppi yokk bu baax limu jëfandikukat yi benn model IA mëna liggéeyal benn yoon.
Lan mooy ñakk kattan gi gëna mag ci batching static (fixe) ngir liggéey LLM?
Ndax guddaayu génne gi dafay wuute, toppalante yi jeex dañuy toog ba keroog bi gëna yeex jeex, yàq cycle GPU yi ak yeexal laajte yu bees yi.
Ci yan granularite la batching biy wéy di yokk wala di dindi ay laaj?
Batching bu wéy (niveau iteration) dafay yeesal ensemble biy dox ginaaw jéego bu nekk ci decode, moo tax dafay dox token-by-token moo gëna nekk ci laaj bi yépp.
Su benn sequence jeexee ci diggu batch bi ci biir batching bu wéy, lan mooy dall slot bi?
Sequences yu jeex yi dañu leen di dàq, ba noppi ñuy dugal ci saasi laaj yi ñuy xaar, batch bi di fees, GPU bi di liggéey.
Ban pexe mooy méngoo bu baax ak batching bu wéy ngir mëna dugal/dindi ay toppalante yu yomb?
PagedAttention dafay denc cache KV bu sekans bu nekk ci ay blok yu moom seen bopp, kon yokk wala dindi sekans ci diggu naaw du yàq memory ñeneen ñi.
Ban këyitu gëstu lañu gëna wax ni moo siiwal batching bi ci niveau iteration (di wéy)?
Këyitu Orca dafa dugal ay jamonoy iteration, ba noppi ñu jël xalaat bi ci sistem yu melni vLLM, TGI, ak TensorRT-LLM.