HejuruUbuyobozi bukurikira
KServe na Model Gukorera Kubernetes
Tekiniki
Ubuyobozi bwa tekiniki
Ubwubatsi bukora bugabanya imvugo nini yerekana urugero mubice bibiri bitandukanye - prefill na decode - ikanabikoresha kubidendezi bitandukanye bya GPU.
Umva iki gitaboAI-byavuzwe · ~2 min
Ntacyo bitwaye kuko ibyo byiciro byombi bifite ubushake buke bwibikoresho, kandi kubahatira kumashini imwe isesagura ubushobozi kandi bikababaza ubukererwe.
Iyo LLM isubije, ikora mubyiciro bibiri. Prefill isoma icyarimwe icyarimwe hanyuma yubake urufunguzo-agaciro (KV) cache; iyi nini nini, ibangikanye, kubara-guturika kuzuza ibice byimibare ya GPU. Decode noneho itanga ibimenyetso kimwekimwe, buri ntambwe isoma cache yose ya KV-yibuka-umurongo-wuzuye, uhuza-kubara. Kwiruka hamwe, prefill ndende ihagarika decode ya buriwese (umutwe-wumurongo uhagarika), kandi gukubita byombi bitera kwivanga. Gutandukana bishyira prefill kuri pisine imwe ya GPU hanyuma ukarangiza kurindi, kwimura cache cache hagati yabo hejuru yimikoranire yihuse nka NVLink cyangwa InfiniBand. Buri kidendezi cyarateguwe kandi gipimwa mu bwigenge, kunoza ibicuruzwa byiza, koroshya umurizo, no kureka ababikora bagakubita igihe-kuri-mbere-ikimenyetso-nigihe-cyo-gusohora-icyerekezo icyarimwe.
Ikiganiro cyunganira kuganira inzira ndende yerekana compte-iremereye ya prefill cluster, hanyuma imigezi isubiza uhereye kumyibutsa-optimiz ya decode cluster kugirango ukomeze kwandika neza.
NVIDIA Dynamo na vLLM bareke abakoresha bashire prefill zitandukanye hamwe na decode yitsinda ryabakozi kugirango guturika kwigihe kirekire ntibishobora guhagarika ibisekuru bikomeza.
Mooncake (ikoreshwa na Kimi ya Moonshot AI) itandukanya prefill na decode kandi ikongeramo pisine yagabanijwe ya KV-cache kugirango igabanye kwishura inshuro nyinshi mubipimo.
Serivisi yo kurangiza-kode yegurira pisine ntoya ya progaramu ya progaramu ngufi na pisine nini ya decode, kubera ko ikiguzi kinini kiva mugutanga ibimenyetso byinshi bisohoka.
Witege ko gutandukana bizaba nkibisanzwe mubikorwa. Sisitemu nka DistServe, Splitwise, na Mooncake yarayamamaye, kandi vLLM na NVIDIA Dynamo ubu ubwato butandukanye. Ubushakashatsi burimo gusunika KV-cache kwimura optimizasiyo, guhuza cache no kongera gukoresha ibyifuzo, imbaraga zo kongera kuringaniza ibipimo bya prefill / decode munsi yimodoka ihindagurika, hamwe no guhuza cyane hamwe na prefix cishing hamwe na prefill chunked prefill. Mugihe imiterere ya Windows ikura miriyoni yikimenyetso, gutandukanya ibi byiciro biba ngombwa cyane kubiciro bikoresha neza, bitinze cyane.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Ubwubatsi bukora bugabanya imvugo nini yerekana urugero mubice bibiri bitandukanye - prefill na decode - ikanabikoresha kubidendezi bitandukanye bya GPU. Ntacyo bitwaye kuko ibyo byiciro byombi bifite ubushake buke bwibikoresho, kandi kubahatira kumashini imwe isesagura ubushobozi kandi bikababaza ubukererwe.
Prefill itunganya ibintu byose muburyo bubangikanye kandi yuzuza compte, mugihe decode isoma cache cache buri ntambwe kandi igarukira kububiko bwibuke-butandukanye nibyifuzo byerekana ibidengeri bitandukanye, byigenga.
Prefill yubaka cache cache kubibazo; decode ikeneye iyo cache kugirango ikomeze kubyara, bityo cache yoherejwe hejuru yihuta ihuza pisine.
Kuri GPU isangiwe birebire biturika birashobora guhagarika intambwe ya decode ikomeza; kubitandukanya birinda ibyo kwivanga no guhagarika ubukererwe bwumurizo.
Kuzuza ibintu byose byihuta hamwe, bityo ibyiciro binini bigaburira tensor cores neza; decode itanga ikimenyetso kimwe icyarimwe kandi igashyirwaho na memoire, bityo igapima ukundi.
Umuyoboro mwinshi, umuyoboro muto-utinda nka NVLink (intra-node) na InfiniBand (inter-node) birakenewe kugirango ihererekanyabubasha rya KV ntirihinduka icyuho gishya.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
KServe na Model Gukorera Kubernetes
Tekiniki