Nchọpụta nhụta anya
Ndozi nhụsianya na-eme ka ụdị asụsụ buru ibu na-ewepụta ederede ngwa ngwa site na iji obere ihe ngosi 'draft' dị ngwa iji maa ọtụtụ akara n'ihu, wee nwee nnukwu ihe nlereanya nyochaa ha niile otu mgbe.
Nchịkọta
It speeds up inference 2-3x with identical output quality.
Ime miri emi
Dị ka ọ na-adịkarị, LLM na-ewepụta ederede otu akara n'otu oge: akara ngosi ọ bụla chọrọ ịgafe n'ihu n'ụzọ zuru oke site na nnukwu ụdị, ị nweghị ike ịmalite nke na-esote ruo mgbe nke ugbu a gwụchara. Nke a na-adị ngwa n'ihi na ọ bụ ebe nchekwa, ọ bụghị ịgbakọ - GPU na-etinye oge ka ukwuu n'ibu ibu, ọ bụghị ime mgbakọ na mwepụ. Ndozi nhụta anya na-agbaji ọkpọ. Obere ihe nlere anya dị ọnụ ala na-atụpụta mkpirisi, sịnụ, akara ngosi ise. Nnukwu ihe atụ 'lekwasịrị' wee hazie ise niile n'otu ngafe na-aga n'ihu wee lelee ha. A na-anabata akara ndị dabara na ihe ọ gaara emepụta; na esemokwu mbụ ọ na-agbazi ma tụfuo ndị ọzọ. N'ihi na ịchọpụta ọtụtụ akara ngosi na-efu ihe dịka ịmepụta otu, ntule anabatara fọrọ nke nta ka ọ bụrụ n'efu.
Nghọta nka nka
Akụkụ amamihe dị na ya bụ usoro nleba anya njụta nke na-ekwe nkwa nkesa mmepụta bụ mgbakọ na mwepụ na-agba ọsọ ụdị naanị - ya mere àgwà adịghị nso, ọ bụ kpọmkwem. Ọnụego nnabata na-akpali ọsọ ọsọ: ka obere ihe nlereanya na-ebu amụma nke buru ibu, ka akara ngosi ndị ọzọ na-arapara n'otu usoro nkwenye. Ụdị dị iche iche dị ka Medusa na-agbakwụnye isi amụma ndị ọzọ na ihe atụ ezubere iche n'onwe ya, yana usoro EAGLE na oghere njirimara, na-ewepụ mkpa maka ụdị nhazi dị iche.
Mmetụta atụmatụ
Ọnụ ego na mmefu ego
Mkpebi ihe owuwu ụlọ na-akwalite arụmọrụ yana ọnụ ahịa ọrụ ruo ọtụtụ afọ.
Mkpebi doro anya
Nkà mmụta nka na-enyere ndị otu egwuregwu aka ịhọrọ nchịkọta ziri ezi, ọ bụghị naanị nke kachasị ọhụrụ.
Quality akara
Nhọrọ injinia ka mma na-ebelata ihe omume ntụkwasị obi na mmepụta.
Ọdịnihu nke Nchọpụta Nkọwapụta
Ndozi nleba anya na-aghọ nke ndabara n'ije ozi dị ka vLLM na TensorRT-LLM. Na-atụ anya ka ụzọ ịdepụta onwe (Medusa, EAGLE, Lookahead) ga-achị ebe ọ bụ na ha na-ezere idowe ihe atụ nke abụọ, gbakwunyere ntule osisi nke na-enyocha ọtụtụ alaka ndị na-eme ntuli aka kwa nzọụkwụ. Ka ụdịdị na-eto eto, mkpọmkpọ ebe nchekwa na-akawanye njọ, na-eme ịkọ nkọ ka ọ baa uru karị, na ndị na-edepụta ngwaike maara ihe ga-eme ka ọsọ ọsọ nke ụwa dị elu.
Mmejuputa n'ezie n'ụwa
Ihe ngosi 7B na-atụ aro akara maka ụdị nkata 70B iji belata nzaghachi nzaghachi na onye enyemaka mmepụta.
Ndị isi Medusa gbadara na LLM ka ọ na-ebu amụma ọtụtụ akara n'ọdịnihu n'otu oge na-enweghị usoro dị iche.
vLLM na-enye nhọpụta ntule iji bulite ntinye nrịbama kwa nkeji nke abụọ na ụyọkọ ozi.
EAGLE na-edepụta n'ime oghere njirimara zoro ezo nke ihe nlereanya ahụ iji kwalite ọnụego nnabata yana ọsọ n'ozuzu ya
Ihe ize ndụ & okporo ụzọ nche
Ịkwalite otu akara ngosi nwere ike zoo adịghị ike sistemụ sara mbara.
A na-eledakarị ihe akụrụngwa na ụgwọ ọrụ anya.
Ọdịiche nchekwa na nleba anya nwere ike itolite ka sistemu na-adịwanye mgbagwoju anya.
Map mmejuputa
Kọwaa latency, ịdịmma na ebumnuche ọnụ ahịa tupu mmejuputa ya.
Benchmark n'okpuru ibu dị adị na ọnọdụ data.
Nleba anya akụrụngwa maka mperi, ịkpafu na mmetụta onye ọrụ.
Kwadebe ụzọ nzaghachi azụghachi azụ na ihe omume tupu ịchachaa.
Nọgide na-eme nchọpụta
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Ntuziaka na-esote
Iji EAGLE emezi ihe nlere anya
Ajụjụ a na-ajụkarị
What is Speculative Decoding?
Ndozi nhụsianya na-eme ka ụdị asụsụ buru ibu na-ewepụta ederede ngwa ngwa site na iji obere ihe ngosi 'draft' dị ngwa iji maa ọtụtụ akara n'ihu, wee nwee nnukwu ihe nlereanya nyochaa ha niile otu mgbe. Ọ na-eme ka inference 2-3x dị elu yana ogo mmepụta yiri ya.
Kedu ihe bụ isi echiche nke nhọpụta ntule?
Ihe nlere ngwa ngwa na-atụ aro ọtụtụ tokens, na nnukwu ụdị ebumnuche na-enyocha ha niile n'otu ngafe yiri ya.
Kedu ihe kpatara ọgbọ LLM nkịtị otu-token n'otu oge ji eji nwayọ?
Nzọụkwụ ọ bụla na-ebunye nnukwu matrices dị arọ site na ebe nchekwa kama ịme mgbakọ dị arọ, yabụ ejighi GPU.
Kedu ihe na-eme na akara nke mbụ ebe usoro na ụdị ebumnuche ekwenyeghị?
A na-anabata akara aka ruo esemokwu ahụ; site na ndakọrịta n'ihu, a jụrụ ha ma debe akara ngosi ziri ezi nke ihe nlereanya ahụ.
Kedu ka nhọpụta ntule si emetụta ogo mmepụta?
Usoro nleba anya njụta na-ekwe nkwa nkesa ikpeazụ dabara na ụdị ebumnuche, yabụ ogo agbanweghị.
Kedu ihe na-ekpebi ngwa ngwa site na nhọpụta ntule?
Ka akara ngosi ndị a na-edepụta nke ọma na-anabata nke ọma n'otu usoro nkwenye, ka ọ na-abawanye ngwa ngwa.