HejuruUbuyobozi bukurikira
Seldon Core hamwe nigishushanyo mbonera
Tekiniki
Ubuyobozi bwa tekiniki
Igenzura rigena icyitegererezo kopi, GPU, cyangwa inyuma bigomba gukemura buri cyifuzo cya LLM cyinjira, nuburyo bwo gukwirakwiza traffic kuburyo nta seriveri imwe irenze.
Umva iki gitaboAI-byavuzwe · ~2 min
Byakozwe neza, bigabanya ubukererwe nigiciro; bikozwe nabi, bitera igihe cyigihe na GPU idafite akazi.
Gukorera LLM mubipimo bisobanura gukoresha kopi nyinshi muri GPU nyinshi, kandi traffic traffic iraturika kandi itaringaniye - ibisubizo biratandukanye cyane muburebure no mubibazo. Router yicaye imbere ahitamo aho yerekeza akoresheje ibimenyetso bikize cyane kuruta classique ya round-robin. Inzira zigezweho za LLM zitekereza ku burebure bwumurongo, KV-cache ituye, kandi niba kopi isanzwe ifite ihuza ryihuta ryibanze (prefix-cache affinity), bityo rero gukurikiranwa bisaba ubutaka aho cache iba. Routers zimwe na zimwe zitoranya icyitegererezo cyo gukoresha-kohereza ibibazo byoroshye kuri moderi ntoya ihendutse kandi ikomeye kuri nini (inzira ya moderi). Kuringaniza imizigo noneho binganya igitutu kuri kopi kugirango wirinde ahantu hashyushye, kubahiriza igipimo cy’ibipimo, no kugumana ubukererwe bwumurizo muke mugihe byinshi byinjira muri rusange no gukoresha GPU.
Ikiganiro cya chatbot gihuza buri kiganiro kuri kopi ifata cache yayo ya KV, bityo rero gukurikiranira hafi gukubita prefix cache hanyuma ugasubiza vuba.
Sisitemu ya RouteLLM yohereza ibibazo byoroshye kuri moderi ntoya ihendutse kandi ikazamura gusa ibikomeye kurwego rwumupaka, kugabanya ibiciro hamwe nigihombo gito.
Kubernetes Gateway API Yerekana inzira Yagutse kubuzima bwa GPU bwimbitse hamwe na cache leta aho kuba umuzenguruko-robin unyuze hejuru.
LiteLLM ihagararira urujya n'uruza kuri OpenAI, Anthropic, hamwe na moderi yakiriye wenyine hamwe no gusubira inyuma no kugereranya igipimo ntarengwa cyo kumenya iyo utanga isoko.
Inzira ihinduka icyiciro-cyambere, cyize. Imishinga nka Kubernetes 'Gateway API Yagutse Yagutse, ibicuruzwa bya vLLM, hamwe na LiteLLM / Intumwa zishingiye kuri routers zisanzwe cache-izi kandi ikanamenya igiciro. Tegereza byinshi byerekana kandi bigoye gushingira kumurongo (RouteLLM-stil), umurongo wa SLA utwarwa numurongo wambere, uturere twinshi hamwe no kumenya-urugero, hamwe na politiki-yize-ishimangira ihuza ubukererwe, ibicuruzwa, hamwe nigiciro cyamadorari mugihe nyacyo nkicyitegererezo, ibiciro, no guhinduranya ibinyabiziga.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Igenzura rigena icyitegererezo kopi, GPU, cyangwa inyuma bigomba gukemura buri cyifuzo cya LLM cyinjira, nuburyo bwo gukwirakwiza traffic kuburyo nta seriveri imwe irenze. Byakozwe neza, bigabanya ubukererwe nigiciro; bikozwe nabi, bitera igihe cyigihe na GPU idafite akazi.
LLM ibyifuzo biratandukanye cyane muburebure / ikiguzi, hamwe na kopi ya KV ya cache ituma amasomo akomera, bityo gusiganwa ku magare buhumyi birengagiza cache isano nuburemere nyabwo.
Niba kopi imaze gufata KV cache kubisanzwe bisangiwe, iyobora ikurikiranwa ryongeye gukoresha iyo cache aho kuyisubiramo, kubika compte nubukererwe.
Routeur yicyitegererezo nka RouteLLM yohereza ibibazo byoroshye kuri moderi ntoya ihendutse kandi ikabika imipaka ihenze yimipaka igoye, igabanya igiciro hamwe nigihombo gito.
Telemetrie nyayo yinyuma-itegereje ibimenyetso, ibyuzuye byuzuye, cache ituye-byerekana umutwaro nyawo neza kuruta kubara byoroshye.
LiteLLM ikora nka porokisi ikurikirana kubatanga (OpenAI, Anthropic, wikorera wenyine), wongeyeho gusubira inyuma no kugereranya igipimo ntarengwa.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Seldon Core hamwe nigishushanyo mbonera
Tekiniki