InoteveraGaidhi rinotevera
KServe uye Model Kushanda paKubernetes
Tekinoroji
Nhungamiro yehunyanzvi
Disaggregated Prefill uye Decode Serving
Chivakwa chinoshandira chinotsemura hombe modhi yemutauro kuita zvikamu zviviri zvakaparadzana-prefill uye decode-uye inovamhanyisa pamadziva akasiyana eGPU.
Teerera kugwaro iriAI-yakarondedzerwa · ~2 min
Pa peji ino2 min verenga
Pfupiso
It matters because these two phases have opposite hardware appetites, and forcing them onto the same machines wastes capacity and hurts latency.
Kudzika Kwakadzika
Kana LLM yapindura, inoshanda mumatanho maviri. Prefill inoverenga kukurumidza kwese kamwechete uye inovaka kiyi-kukosha (KV) cache; uku kukuru, kwakafanana, kuputika-kusungwa kwekombuta kunozadza zvikamu zvemasvomhu zveGPU. Decode yobva yagadzira tokeni imwe panguva, nhanho imwe neimwe ichiverenga yese KV cache-yekuyeuka-bandwidth-yakasungwa, zvishoma-compute trickle. Mhanyai pamwe chete, refu prefill inomisa decode yemunhu wese (musoro-we-mutsara kuvharira), uye batching izvo zviviri zvinogadzira kukanganisa. Disaggregation inoisa prefill pane imwe GPU dziva uye decode pane imwe, kuendesa iyo KV cache pakati payo pamusoro inokurumidza kubatanidza seNVLink kana InfiniBand. Dziva rega rega rakarongedzerwa uye kuyerwa rakazvimiririra, kuvandudza goodput, kutsetseka muswe latency, uye kuita kuti vashandisi varove zvakasimba nguva-kusvika-yekutanga-chiratidzo uye nguva-yega-yekubuda-chiratidzo chetariro panguva imwe chete.
Kuitwa muNyika yeChokwadi
Mubatsiri wechat anoendesa gwaro refu richienda kune compute-inorema prefill cluster, ipapo nzizi mhinduro kubva mu memory-optimized decode cluster kuti kutaipa kurambe kwakapfava.
NVIDIA Dynamo uye vLLM regai vashandisi vatumire zvakaparadzana prefill uye decode mapoka evashandi kuitira kuti kuputika kwekureba kurege kuomesa zvizvarwa zvirikuenderera mberi.
Mooncake (inoshandiswa neMoonshot AI's Kimi) inopatsanura prefill uye decode uye inowedzera yakagoverwa yeKV-cache dziva kucheka kukurumidza kudzokororwa pachiyero.
Iyo kodhi-yekupedzisa sevhisi inotsaurira diki prefill dziva rekukurumidza kukurumidza uye hombe decode dziva, sezvo mari zhinji inobva mukutepfenyura akawanda anobuda tokeni.
Ramangwana reDisaggregated Prefill uye Decode Serving
Tarisira kuti disaggregation ive yekusagadzika mumatura ekugadzira. Masisitimu akaita seDistServe, Splitwise, uye Mooncake akasimudzira, uye vLLM neNVIDIA Dynamo ikozvino inotumira nzira dzakapatsanurwa. Tsvagiridzo iri kusundira KV-cache kutamisa optimizations, cache kubatanidza uye kushandisa zvakare pane zvese zvikumbiro, zvine simba kuenzanirana kweprefill/decode reshiyo pasi pekuchinja kwetraffic, uye kusanganisa kwakasimba ne prefix caching uye chunked prefill. Sezvo mahwindo emamiriro anokura kuita mamirioni ezviratidzo, kupatsanura zvikamu izvi kunowedzera kukosha kune inodhura-inoshanda, yakaderera-latency kushumira.
Kuzvishandisa
Nei zvichikosha
- Mutengo uye bhajeti. Zvisarudzo zvezvivakwa zvinotyaira kuita uye mutengo wekushandisa kwemakore.
- Sarudzo dzakajeka. Dzidzo yehunyanzvi inobatsira zvikwata kusarudza murwi wakakodzera, kwete iwo mutsva chete.
- Kudzora kwemhando yepamusoro. Sarudzo dzeinjiniya dziri nani dzinoderedza zviitiko zvekuvimbika mukugadzira.
Chenjera
- Kugadzirisa imwe bhenji kunogona kuvanza yakafara system kushaya simba.
- Infrastructure uye mari yekugadzirisa inowanzotarisirwa pasi.
- Chengetedzo uye kucherechedzwa mapundu anogona kukura sezvo masisitimu anowedzera kuoma.
Matanho anotevera
- Tsanangura latency, mhando, uye mutengo zvinangwa usati waitwa.
- Benchmark pasi pechokwadi mutoro uye data mamiriro.
- Chishandiso chekutarisa zvikanganiso, kudonha, uye mushandisi maitiro.
- Gadzirira nzira dzekudzosera kumashure uye dzezviitiko usati wawedzera.
Ramba Uchiongorora
Tsamba yemahara
Tora iyo yemazuva ese AI pfupiso
Nhatu dzakasimbiswa dzeAI nyaya mangwanani ega ega pakati pevhiki, dzakanyorwa neChirungu chakajeka. Yemahara zvachose, hapana ads.
Imwe email zuva rega rega revhiki. Bvisa kunyoresa kamwe chete. Hatimbotengese kana kugovera kero yako.
Zviedze iwe pachako
Take the Disaggregated Prefill and Decode Serving quiz
Mhinduro yepakarepo pamhinduro yega yega, uye chitupa chinogovaniswa chine ID inobatika kana wapedza kosi.
Tsigira dzidzo yemahara yeAI. AI Understanding ndeye 501(c)(3) isingabatsiri - hapana ads, hapana paywall, zvachose. Ita chipo
Mibvunzo inowanzo bvunzwa
What is Disaggregated Prefill and Decode Serving?
Chivakwa chinoshandira chinotsemura hombe modhi yemutauro kuita zvikamu zviviri zvakaparadzana-prefill uye decode-uye inovamhanyisa pamadziva akasiyana eGPU. Izvo zvine basa nekuti zvikamu zviviri izvi zvine zvakapesana nezvishuwo zvehardware, uye kuvamanikidza kuti vapinde mumichina imwechete kutambisa huwandu uye kukuvadza latency.
Ndechipi chikonzero chehardware chikonzero chekuparadzanisa prefill uye decode pamadziva eGPU akasiyana?
Prefill inogadzirisa iyo yese kukurumidza mukufanana uye inozadza compute, nepo decode ichiverenga iyo KV cache nhanho imwe neimwe uye inoganhurwa nendangariro bandwidth-inopikisa zvishuwo zvinoruramisa akaparadzana, akazvimirira madziva akarongeka.
Ndechipi chimiro chedata chinofanira kutamiswa kubva kuvashandi vanofanozadza kuti vatore vashandi?
Prefill inovaka iyo KV cache yekukurumidza; decode inoda iyo cache kuti ienderere mberi nekugadzira, saka cache inotumirwa pamusoro pekukurumidza kubatanidzwa kune decode dziva.
Nderipi dambudziko iro disaggregation inoderedza zvakanyanya mukugovaniswa-GPU setup?
PamaGPU akagovaniswa kureba kwekuzadza kwekutanga kunogona kuvharira kuenderera mberi decode matanho; kuvaparadzanisa kunodzivirira kupindira ikoko uye kunodzikamisa muswe latency.
Nei prefill ichigona kuunganidzwa zvine hukasha asi decode mabhenefiti kubva kune akasiyana tuning?
Prefill maitiro ese ekukurumidza tokens pamwechete, saka mabheji akakura anodyisa ma tensor cores zvakanaka; decode inogadzira chiratidzo chimwe panguva uye inovharwa nendangariro, saka inoyera zvakasiyana.
Ndedzipi dzekubatanidza dzinowanzo shandiswa kufambisa iyo KV cache pakati pemadziva akapatsanurwa?
Yakakwira-bandwidth, yakaderera-latency zvinongedzo seNVLink (intra-node) uye InfiniBand (inter-node) inodiwa kuitira kuti KV-cache kutamisa kusave iyo nyowani bhodhoro.
Ramba uchidzidza
Mazano anoenderana
Mamwe madhairekitori akasarudzirwa nyaya iyi