Batching na-aga n'ihu
Batching na-aga n'ihu bụ usoro ọrụ na-agbakwunye ma na-ewepụ arịrịọ sitere na token-token na-agba ọsọ, kama ichere ogbe edoziri ka ọ mechaa.
Nchịkọta
Ọ na-eme ka GPU na-arụsi ọrụ ike mgbe niile ma na-abawanye ọnụ ọgụgụ ndị ọrụ ụdị AI nwere ike ije ozi n'otu oge.
Ime miri emi
GPU na-aka ngwa ngwa mgbe ha na-ahazi ọtụtụ arịrịọ ọnụ na otu. Ụzọ nzuzu, batching static, na-achịkọta arịrịọ a kapịrị ọnụ, na-eme ha niile ka emechaa, wee malite ogbe na-esote. Nsogbu a: mmepụta ụdị asụsụ dịgasị iche n'ogologo, ya mere arịrịọ dị mkpirikpi na-agwụ n'isi ụtụtụ na oghere ha na-anọdụ ala na-adịghị arụ ọrụ mgbe batch na-echere nke kachasị ogologo, na-emebi usoro GPU na-egbu oge na-abịarute ọhụrụ. Batching na-aga n'ihu (nke a na-akpọkwa n'ime ụgbọ elu ma ọ bụ ọkwa ọkwa iteration, nke akwụkwọ Orca na-ewu ewu ma jiri ya na vLLM, TensorRT-LLM, na TGI) na-arụ ọrụ n'ogo nke otu nzọụkwụ nhazi. Mgbe emechara akara ọ bụla, usoro emechachara pụọ na batch ahụ wee tinye arịrịọ ndị bịarutere ọhụrụ ozugbo. Nke a na-eme ka ogbe ahụ zuo ezu na GPU juputara, na-abawanye mmepụta ugboro ugboro na obere latency maka ndị ọrụ na-echere.
Nghọta nka nka
Isi ngbanwe bụ site n'ịchịkọta arịrịọ niile ruo n'ịchịkọta mmegharị nke onye ọ bụla. Na usoro ngbanwe ọ bụla, onye nhazi oge na-ewulite ihe arụrụ arụ ọrụ: ọ na-agba ọsọ otu ụzọ gafere n'usoro ụgbọ elu niile, na-ewepụta otu akara nke ọ bụla, na-achụpụ ihe ọ bụla kụrụ akara njedebe ma ọ bụ oke ogologo, wee kweta arịrịọ kwụ n'ahịrị iji mejupụta oghere ndị a tọhapụrụ. Ijikọ nke a na ebe nchekwa KV na-agbanwe agbanwe na PagedAttention na-eme ntinye na iwepu usoro n'etiti ụgbọ elu dị ọnụ ala, ebe nchekwa nke usoro nke ọ bụla na-ebi na ngọngọ nke onwe.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke batching na-aga n'ihu
Batching na-aga n'ihu ugbu a bụ ọkọlọtọ na mmepụta LLM. Ọrụ ga-eme n'ọdịnihu na-emezi onye nhazi oge: ikewapụta usoro njuju-mgbakọ dị arọ na usoro ngbanwe dị mfe (nkewa), prefill chunked iji zere ngbanwe ngbanwe, atumatu mkpa na izi ezi maka ibu ọrụ agwakọta, yana njikọta siri ike na ntule ntule nke mere na a na-akwado ọtụtụ akara ngosi otu nzọụkwụ. Ebumnobi a bụ ịpịnye akara kacha elu-kwa-sekọnd kwa GPU ka ị na-eme ka nzaghachi onye ọ bụla dị ala na nke a ga-ebu amụma.
Mmejuputa n'ezie n'ụwa
API nkata na-anabata ozi ndị ọrụ bịarutere ọhụrụ n'ime batch na-agba ọsọ ozugbo kama ị kwụ ha n'ahịrị maka ogbe na-esote.
Ịchụpụ azịza dị mkpirikpi emechara n'etiti ogbe ma mejupụta oghere ya ka GPU ghara ịdị na-eche ogologo ọgbọ.
Na-ejikọta batching na-aga n'ihu na vLLM's Paged Ntị iji tinye ma wepụ usoro dị ọnụ ala na usoro nbibi ọ bụla
Ọrụ mmecha koodu na-akwado akara ngosi dị elu-kwa-sekọnd n'okpuru gbawara agbawa, okporo ụzọ ogologo na-agbanwe agbanwe site na idobe ogbe ahụ n'uju.
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Continuous Batching quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Kubernetes maka ibu ọrụ ML
Ajụjụ a na-ajụkarị
Kedu ihe bụ Batching Na-aga n'ihu?
Batching na-aga n'ihu bụ usoro ọrụ na-agbakwunye ma na-ewepụ arịrịọ sitere na token-token na-agba ọsọ, kama ichere ogbe edoziri ka ọ mechaa. Ọ na-eme ka GPU na-arụsi ọrụ ike mgbe niile ma na-abawanye ọnụ ọgụgụ ndị ọrụ ụdị AI nwere ike ije ozi ozugbo.
Gịnị bụ isi adịghị ike nke static (obere) batching maka ozi LLM?
N'ihi na ogologo mmepụta na-adịgasị iche, usoro emechara na-anọdụ ala na-abaghị uru ruo mgbe onye kacha dị nwayọọ mezuru, na-emebi okirikiri GPU na-egbu oge arịrịọ ọhụrụ.
Kedu granularity na-aga n'ihu na-agbakwunye ma wepụ arịrịọ?
Ọkwa na-aga n'ihu (iteration-level) na-emelite ntọala nọ n'ọrụ ka emechara usoro ngbanwe ọ bụla, yabụ ọ na-arụ ọrụ token site-token karịa arịrịọ niile.
Mgbe usoro mechara n'etiti ogbe n'okpuru batching na-aga n'ihu, gịnị na-eme oghere ya?
A na-achụpụ usoro emechachara wee tinye arịrịọ nchere ozugbo, na-eme ka ogbe ahụ juputara na GPU na-arụ ọrụ.
Kedu usoro na-ejikọta ya na batching na-aga n'ihu iji mee ka ntinye/iwepụ usoro dị ọnụ ala?
PagedNlebara anya na-echekwa oghere KV nke usoro nke ọ bụla n'ime ngọngọ nke onwe ya, yabụ ịgbakwụnye ma ọ bụ iwepu usoro n'etiti ụgbọ elu anaghị emebi ebe nchekwa ndị ọzọ.
Kedu akwụkwọ nchọcha ka a na-ekwukarị na ọ bụ ọkwa ọkwa na-aga n'ihu (na-aga n'ihu) batching?
Akwụkwọ Orca webatara nhazi ọkwa ọkwa iteration, wee nakweere echiche ahụ site na sistemụ ozi dị ka vLLM, TGI, na TensorRT-LLM.