Gukomeza
Gukomeza guterana ni tekinike yo gutanga yongeramo kandi ikuraho ibyifuzo bivuye mukiruka cyerekanwa-by-token, aho gutegereza icyiciro cyose cyarangiye.
Incamake
It keeps the GPU constantly busy and sharply increases how many users an AI model can serve at once.
Kwibira cyane
GPU irihuta cyane mugihe itunganya ibyifuzo byinshi hamwe mugice. Uburyo bwa naive, guhuza static, amatsinda ahamye yo gusaba, akayobora byose kugeza arangije, hanyuma atangira icyiciro gikurikira. Ikibazo: imvugo yicyitegererezo yindimi ziratandukanye cyane muburebure, kubwibyo gusaba bigufi birangira hakiri kare kandi ibibanza byabo bicara ubusa mugihe icyiciro gitegereje igihe kirekire, guta GPU cycle no gutinza abaje bashya. Gukomeza guhora (nanone byitwa mu ndege cyangwa itera-urwego rwo guterana, bikunzwe nimpapuro za Orca kandi bikoreshwa muri vLLM, TensorRT-LLM, na TGI) bikorera kuri granularite yintambwe imwe ya decoding. Nyuma ya buri kimenyetso kimaze gukorwa, urutonde rwarangije gusohoka mucyiciro kandi ibyifuzo bishya byahise byerekanwa ako kanya. Ibi bituma icyiciro cyuzuye kandi GPU yuzura, akenshi bizamura ibicuruzwa inshuro nyinshi hejuru hamwe nubukererwe buke kubakoresha gutegereza.
Ubushishozi
Ihinduka ryibanze ni uguhuza ibyifuzo byose kugeza kubisubiramo kugiti cye. Kuri buri ntambwe ya decode gahunda yubaka igenamigambi ikora: ikoresha inzira imwe yimbere hejuru yindege zose zikurikirana, ikohereza ikimenyetso kimwekimwe, ikirukana icyaricyo cyose cyakubise iherezo ryikurikiranya cyangwa uburebure, kandi ikemera ibyifuzo byateganijwe kugirango byuzuze ibibanza byarekuwe. Guhuza ibi hamwe na PagedIbyitonderwa byoroshye KV yibuka ituma winjiza kandi ukuraho urutonde rwagati rwagati ruhendutse, kubera ko buri cache ikurikirana iba mubice byigenga.
Ingaruka z'Ingamba
Igiciro na bije
Ibyemezo byubwubatsi bitwara imikorere nigiciro cyimikorere kumyaka.
Ibyemezo bisobanutse
Ubuhanga bwa tekinike bufasha amakipe guhitamo umurongo ukwiye, ntabwo ari shyashya gusa.
Kugenzura ubuziranenge
Guhitamo neza bya injeniyeri bigabanya ibintu byizewe mubikorwa.
Igihe kizaza cyo gukomeza gufata
Gukomeza guterana ubu nibisanzwe mubikorwa LLM itanga. Akazi kazoza gatunganya gahunda: gutandukanya compte-iremereye yicyiciro cyambere nicyiciro cyoroshye cya decode (kutavuga rumwe), prefill yuzuye kugirango wirinde guhagarara decoding, politiki yibanze hamwe nuburinganire kubikorwa bivanze nakazi kavanze, hamwe no gukomera hamwe na decoding yibitekerezo kuburyo ibyemezo byinshi byerekana byemewe kuri buri ntambwe. Intego ni ugusunika ibimenyetso ntarengwa-isegonda kuri GPU mugihe kugumya gutinda kugiti cyawe hasi kandi byateganijwe.
Gushyira mu bikorwa Isi
Ikiganiro API cyemera ubutumwa bwabakoresha bashya mugihe cyo kwiruka ako kanya aho kubatonda umurongo mugice gikurikira
Kwirukana igisubizo kigufi cyuzuye hagati yicyiciro no gusubiza inyuma aho GPU itigera ikora idategereje ibisekuru birebire
Gukomatanya guhoraho hamwe na vLLM's PagedIcyifuzo cyo gushiramo no gukuraho urutonde ruhendutse kuri buri ntambwe ya decode
Serivisi yo kurangiza-kode ikomeza ibimenyetso byinshi-isegonda munsi yaturika, impinduka-ndende yimodoka ukomeza icyiciro cyuzuye
Ingaruka & Kurinda
Gutezimbere igipimo kimwe gishobora guhisha intege nke za sisitemu.
Ibikorwa Remezo no kubungabunga akenshi usanga bidahabwa agaciro.
Icyuho cyumutekano no kwitegereza birashobora kwiyongera uko sisitemu igenda igorana.
Igishushanyo mbonera
Sobanura ubukererwe, ubuziranenge, nigiciro cyibiciro mbere yo kubishyira mubikorwa.
Ibipimo byerekana umutwaro ufatika hamwe namakuru yimiterere.
Gukurikirana ibikoresho kubikosa, drift, ningaruka zabakoresha.
Tegura inzira yo gusubiza ibyabaye mbere yo gupima.
Komeza Ubushakashatsi
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continuous Batching quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ubuyobozi bukurikira
Kubernetes ya ML Imirimo Yakazi
Ibibazo bikunze kubazwa
What is Continuous Batching?
Gukomeza guterana ni tekinike yo gutanga yongeramo kandi ikuraho ibyifuzo bivuye mukiruka cyerekanwa-by-token, aho gutegereza icyiciro cyose cyarangiye. Bituma GPU ihora ihuze kandi ikongera cyane umubare wabakoresha moderi ya AI ishobora gukorera icyarimwe.
Nubuhe ntege nyamukuru zo gutondeka static (zihamye) zo gukorera LLM?
Kuberako ibisohoka birebire biratandukanye, ibyarangiye bikurikiranye bicara ubusa kugeza igihe bitinze birangiye, guta GPU cycle no gutinza ibyifuzo bishya.
Ni ubuhe bwoko bwa granularitike guhora wongeyeho kandi ukuraho ibyifuzo?
Gukomeza (itera-urwego) kuvugurura bigezweho gushiraho nyuma yintambwe imwe ya decode, bityo ikora token-by-token kuruta kubisabwa byose.
Iyo urukurikirane rurangije hagati yicyiciro gikomeza, bigenda bite kumwanya wacyo?
Urutonde rwarangiye rwirukanwe kandi ibyifuzo byo gutegereza birahita byinjira, bikomeza icyiciro cyuzuye kandi GPU irahuze.
Ni ubuhe buryo bukoreshwa muburyo busanzwe hamwe no gukomeza guterana kugirango ushiremo / ukureho ibihe bihendutse?
Urupapuro rwitondewe rubika buri cyiciro cya KV cache muburyo bwigenga, bityo rero kongeraho cyangwa gukuraho urutonde rwindege rwagati ntibibuza abandi kwibuka.
Ni izihe mpapuro z'ubushakashatsi zisanzwe zishimirwa no gukwirakwiza itera-urwego (rukomeza)?
Urupapuro rwa Orca rwerekanye gahunda yo gutondekanya urwego, kandi igitekerezo cyakiriwe mugukorera sisitemu nka vLLM, TGI, na TensorRT-LLM.