Speculative Iyipada Akọpamọ Models
Iyipada asọye n lo awoṣe 'akọpamọ' kekere kan, iyara lati gboju ọpọlọpọ awọn ami ti n bọ pe awoṣe nla lẹhinna jẹrisi ni igbasilẹ kan.
Akopọ
O ṣe iyara iran ọrọ 2-3x laisi iyipada si abajade.
Jin Dive
Awọn awoṣe ede ti o tobi n ṣe agbejade ọrọ ami ami kan ni akoko kan, ati pe igbesẹ kọọkan nilo gbigbe siwaju ni kikun nipasẹ awọn ọkẹ àìmọye awọn ayeraye - o lọra ati-iwọn iranti. Speculative decoding attacks this by pairing the big 'target' model with a cheap 'draft' model. The draft model rapidly proposes a chunk of, say, 4-8 candidate tokens. The big model then processes all of them in a single parallel forward pass and checks each one. Tokens that match what the big model would have produced are accepted; the first mismatch is corrected and the rest discarded. Because verifying several tokens at once costs roughly the same as generating one, accepted runs are nearly free. Ni pataki, igbesẹ iṣapẹẹrẹ ijusile ṣe iṣeduro pinpin ipari jẹ aami kanna si ṣiṣe awoṣe nla nikan - iyara laisi pipadanu didara.
Imọ-imọ-ẹrọ
The key trick is a modified rejection-sampling test. For each drafted token, the target model's probability is compared to the draft model's. If the target assigns equal or higher probability, the token is accepted; bibẹkọ ti o ti wa ni gba pẹlu iṣeeṣe dogba si awọn ipin, ati lori ijusile ti wa ni titunse àmi ti wa ni apere lati ẹya titunse iyokù. This math makes the output provably equivalent to sampling directly from the large model.
Ipa Ilana
Iyara ati iwọn
Ṣiṣan iṣẹ ede le gbe ni iyara laisi irubọ aitasera.
Wiwọle ati arọwọto
O faagun iraye si kọja awọn ede ati awọn aza ibaraẹnisọrọ.
Awọn ipinnu diẹ sii
Awọn ẹgbẹ le lo akoko diẹ sii lori idajọ lakoko ti adaṣe n kapa atunwi.
Ọjọ iwaju ti Awọn awoṣe Akọpamọ Iyipada Asọtẹlẹ
Expect draft models to become standard infrastructure in inference servers like vLLM and TensorRT-LLM. Self-speculation variants (Medusa, EAGLE) drop the separate draft model entirely by adding lightweight prediction heads, and tree-based drafting verifies many candidate continuations at once. Bi awọn window ti o tọ ti ndagba ati awọn idiyele iṣẹ ṣiṣe jẹ gaba lori, ijafafa, awọn olupilẹṣẹ ti o baamu awoṣe ati iṣeduro imọ-ohun elo yoo Titari awọn oṣuwọn gbigba ati gbigbejade ga julọ.
Real-World imuse
Anthropic, OpenAI, ati Google lo iyipada arosọ lati ge idaduro ati iye owo iṣẹ lori awọn oluranlọwọ iwiregbe ti n sin awọn miliọnu awọn olumulo.
vLLM ati NVIDIA TensorRT-LLM ọkọ oju omi ti a ṣe sinu asọye asọye ki awọn alalejo ti ara ẹni le mu iyara Llama tabi awọn imuṣiṣẹ Mistral.
Pipọpọ awoṣe iyaworan 7B pẹlu ibi-afẹde 70B (fun apẹẹrẹ, idile Llama-3) si aijọju ilọpo meji awọn ami-fun-keji lori GPU kan.
Awọn irinṣẹ ipari koodu lo awoṣe iyaworan kekere kan lati dabaa igbomikana ti awoṣe ti o tobi julọ jẹri, ti o jẹ ki awọn didaba jẹ didan ninu olootu.
Awọn ewu & Awọn ọna iṣọ
Awọn otitọ ti a sọ di mimọ le tẹ awọn ijabọ sii ni idakẹjẹ, awọn ṣiṣan atilẹyin, tabi awọn abajade iwadii.
Ifamọ kiakia le ṣẹda awọn abajade aisedede kọja awọn ibeere ti o jọra.
Awọn data ọrọ ifarabalẹ le farahan ti awọn idari wiwọle ko lagbara.
Ilana Ilana imuse
Ṣetumo ọna kika iṣẹjade, ohun orin, ati awọn iṣedede didara ṣaaju ṣiṣejade.
Awọn idahun ilẹ pẹlu awọn orisun ti o gbẹkẹle nigbakugba ti deede ba ṣe pataki.
Jeki aaye ayẹwo atunyẹwo eniyan fun awọn abajade ti o ga julọ.
Tọpinpin awọn ilana ikuna ati tunṣe awọn itọsi tabi ṣiṣan iṣẹ nigbagbogbo.
Tesiwaju Ṣiṣawari
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding Draft Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Itọsọna atẹle
Awọn atunṣe akiyesi fun Awọn awoṣe koodu
Awọn ibeere ti a beere nigbagbogbo
Kini Speculative Decoding Draft Models?
Iyipada asọye n lo awoṣe 'akọpamọ' kekere kan, iyara lati gboju ọpọlọpọ awọn ami ti n bọ pe awoṣe nla lẹhinna jẹrisi ni igbasilẹ kan. O ṣe iyara iran ọrọ 2-3x laisi iyipada si iṣelọpọ.
Kini ipa akọkọ ti awoṣe 'akọpamọ' ni iyipada arosọ?
Awoṣe apẹrẹ kekere ti o ṣaroye awọn ami ti n bọ, eyiti awoṣe ibi-afẹde nla lẹhinna ṣayẹwo ni iwe-iwọle afiwera kan ṣoṣo.
Kini idi ti ijẹrisi ọpọlọpọ awọn ami kikọ silẹ ni ẹẹkan fi akoko pamọ?
Iran jẹ iranti-owun; Ṣiṣayẹwo awọn ami-ami pupọ ni iwe-iwọle batched kan fẹrẹ jẹ olowo poku bi igbesẹ kan, nitorinaa awọn ṣiṣe ti o gba jẹ ọfẹ ọfẹ.
Kini yoo ṣẹlẹ si awọn ami ti a ti kọ lẹhin ami akọkọ ti awoṣe afojusun kọ?
Gbigbawọle n tẹsiwaju titi ariyanjiyan akọkọ; ti o àmi ti wa ni atunse ati gbogbo awọn tetele drafted àmi ti wa ni da àwọn kuro.
Bawo ni iyipada akiyesi ṣe rii daju pe didara iṣelọpọ ko bajẹ?
Idanwo ijusile-iṣapẹrẹ ti a tunṣe ṣe iṣeduro awọn ibaamu pinpin ami ami ikẹhin ti iṣapẹẹrẹ taara lati awoṣe ibi-afẹde.
Ewo ninu iwọnyi jẹ ọna ‘arosọ ti ara ẹni’ ti o yago fun awoṣe iyaworan lọtọ?
Medusa ati EAGLE ṣafikun awọn olori asọtẹlẹ afikun iwuwo fẹẹrẹ si awoṣe akọkọ ki o le ṣe awọn ami ami iwaju tirẹ.