Dekodage spekulatif
Decodage speculatif dafay tax modeli làkk yu mag yi gëna gaaw defar mbind ci jëfandikoo modelu 'projet' bu ndaw te gaaw ngir xayma ay token yu bari ci kanam, ba noppi model bu mag bi xool leen ñépp benn yoon.
Résumé
It speeds up inference 2-3x with identical output quality.
Plongeur bu xóot
Daanaka LLM dafay defar benn mbind benn token benn yoon: token bu nekk dafay laaj ñu jàll ci kanam bu mat sëkk ci model bu mag bi, te mënoo tàmbali beneen bi ba keroog bi fi nekk jeex. Lii yeexna ndax dafay tënku ci memory, nekkul ci xayma - GPU dafay dundal jot gi gëna bari ci yobbu ay poid, du def math. Decodage speculatif dafay dindi jafe-jafe yi. Benn xeetu projet bu ndaw te yomb dafay propoze benn wàll, nu wax, juróomi jeton kàndidaa. Modèle 'target' bu mag bi dafay jëfandikoo juróom yi ci benn yoon paralel ci kanam ba noppi xool leen. Token yu méngoo ak limu wara defar lañuy nangu; bu njëkkee ñàkka deggoo, dafay jubbanti ba noppi sànni leneen li. Ndax firndeel token yu bari lu tollu ci benn njëg ak defar benn, guesses nangu daanaka amul fayda.
Gis-gis xarala
Wàll wi am xel mooy sàrtu bañ-sampling biy garanti ni distribution biy génne dafay nuru ci wàllu math ak doxal model bi ñuy wut kese - kon kalite bi nekkul luy jege, muy gëna jubal. Taxawam nangu gi mooy dawal gaawaay bi: lu model bu ndaw bi di gëna mëna wax luy waaja am ci bu mag bi, lu gëna bari ay token yuy tëye ci jéego bu nekk ngir xool. Variante yu melni Medusa dañuy yokk yeneen boppu prediction ci model bi ñuy wut ci boppam, ak EAGLE di defar ci espace feature, dindi soxla modelu projet bu wuute.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu dekodage spekulatif
Dekodage spekulatif mingi nekk luñuy jëfandikoo ci stack yu melni vLLM ak TensorRT-LLM. Xaarandil pexe yi ñuy bindee seen bopp (Medusa, EAGLE, Lookahead) ngir ëpp doole ndax dañu moytu tëye ñaareelu xeetu, boole ci xalaat yu sukkandiko ci garab yuy firndeel ay car yu bari ci jéego bu nekk. Lu model yi di màgg, memory-bound bottleneck bi dafay gëna tar, tax speculation bi gëna am solo, te ñiy bind ci hardware bi dina ñu gëna xëcc gaawaayu àdduna dëgg.
Doxal ci àdduna dëgg
Benn xeetu projet bu 7B di proposé jetons ngir benn xeetu chat 70B ngir dagg latency tontu ci benn assistant production
Medusa boppu boulé ci benn LLM suko defee mu wax luy am ci ëlëg jetons yu bari benn yoon te amul benn xeetu projet bu wuute
vLLM may dekodage speculatif ngir yokk produit tokens ci segond bu nekk ci cluster buy liggéey
EAGLE dafay defar ci barabu man-man yu nëbbu yi ngir yokk limu nangu ak gaawaay gu mat sëkk
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Decodage spekulatif ak EAGLE
Laaj yi ñuy faral di laaj
What is Speculative Decoding?
Decodage speculatif dafay tax modeli làkk yu mag yi gëna gaaw defar mbind ci jëfandikoo modelu 'projet' bu ndaw te gaaw ngir xayma ay token yu bari ci kanam, ba noppi model bu mag bi xool leen ñépp benn yoon. Dafay gaawlu inference 2-3x ak kalite génne bu nuróo.
Lan mooy xalaat bu mag bi ci dekodage spekulatif?
Benn modelu projet bu gaaw dafay proposer ay token yu bari, te modelu target bu mag bi dafay saytu leen ñépp ci benn pass parallèle.
Lan moo waral benn token ci benn yoon LLM di yeex?
Jéego bu nekk dafay charger matrices yu diis yu bari yi ci mémoire bi moo gën def calcul yu diis, moo tax GPU bi dañu koy jëfandikoo bu baax.
Lan mooy xew ci token bu njëkk bi nga xamni modelu projet bi ak target bi bokku ñu?
Token yu ba ci ñàkka déggoo lañuy nangu; lu ko dalee ci mismatch bi, dañu leen di bañ, ba noppi ñu denc token bu dëggu bi ci model bi ñuy wut.
Ak naka la dekodage speculatif di amee ci kalite génne gi?
Regle bu bañ-échantillonnage dafay garanti ni séddale bu mujj bi méngoo ndànk ak model biñ bëgga, kon kalite bi du soppeeku.
Lan mooy gëna mëna wax ci gaawaayu dekodage speculatif bi?
Lu gëna bari ay token yuñ bind model biñ bëgga nangu ci jéego bu nekk ngir xool, lu gaawaay bi di gëna rëy.