Decodage spekulatif ak EAGLE
Decodage speculatif dafay gaawlu inference model làkk bu mag ci bàyyi model projet bu ndaw mu xalaat ay token yu bari ci kanam, model bu mag bi dafay xool ci benn yoon.
Résumé
EAGLE is a state-of-the-art version that drafts at the feature level rather than the token level, delivering 2-4x speedups with zero loss in output quality.
Plongeur bu xóot
LLM generation bu jaar yoon mooy autoregressive: model bi dafay defar benn token, mu delloo ko, ba noppi baamtu ko, kon token bu nekk dafay laaj jàll bu mat sëkk ci ay miliyaar ciy parametre. Decodage speculatif mooy dindi jafe-jafe bii. Ab defarkat bu yomb dafay proposer ay token candidat, ak model target bu seer bi dafay xool ñoom ñépp ci benn pass parallèle, nangu prefix bi gëna gudd te jub. EAGLE (Algorithme Extrapolation ngir gëna am njariñ ci modelu làkk) dafa gëna suqaliku ci anam yu njëkk ya, ci defar ci barabu man-mani bu nëbbu bi ci model bi, ba noppi di joxewaat token bi njëkk ci dëgg ngir wàññi ñàkka wóor. EAGLE-2 dafay yokk garabu bordure buy soppiku, EAGLE-3 dafay dindi benn tënk buy wax luy waaja am ngir gëna mëna eskale. Li gëna am solo mooy, xool ndax génne gi dafay nuru li model biñ bëgga génne kese.
Gis-gis xarala
EAGLE dafay tàggat benn boppu autoregressive bu ndaw buy wax luy xew ci model bi ñuy wut, ba noppi jëfandikoowaat boppu LM bi ci boppam ngir soppi man-man yi nekk kàndidaa token. Suñu ko conditionee ci toppalante token biñ toxal ak man-man yi ko jiitu, dafay dagg jaxasoo bi daan sonal man-man yi kese. garabu kàndidaa yi dañu leen xool ci saasi; distribution model biñ bëgga am dañu koy denc ndax jeton yiñ nangu dañu wara méngoo ak tànneef biñ jël wala argmax, suko defee gaawaay bi du ñàkk.
njeextalu pexe
Njëgg ak budget
Dogal yi architecture di jël dañuy indi njariñ ak njëgu liggéey bi ay at ci ginaaw.
dogal yu gëna leer
Njàngalem xarala yi dafay jàppale ekip yi ñu tànn li gën, te baña yam ci li gëna bees daal.
Xool kalite
Tanneef yu gëna baax ci wàllu ingeñër dina wàññi jafe-jafe yi ci wàllu wóor ci liggéey bi.
Ëlëgu dekodage spekulatif ak EAGLE
Dekodage spekulatif mingi nekk jumtukaay biñ jagleel ñiy liggéey ci stack yu melni vLLM ak TensorRT-LLM. Xaarandil lëkkaloo bu gëna dëgër ak batching ak KV-cache séddoo, xeetu self-drafting te soxlawul drafter bu wuute, ak co-design hardware buy xalaat xool paralel. EAGLE-style feature drafting ñu ngi koy yokk ci multimodal ak model xalaat, fu chaine yu guddu xalaat def njëgu token bu nekk lu metti, ak ci inference ci aparey fu latency gëna am solo.
Doxal ci àdduna dëgg
Dagg latency ci assistant chat yi suko defee tontu yi di gëna gaaw 2-3x te doo soppi tontu model bi
Wàññi njëgu serwiis GPU ngir fournisseur API yu bari yi ci defar ay token yu bari ci paas bu nekk
Gaawaale xeetu xalaat yu gudd yi, fu ñuy defaree ay junni token ci laaj bu nekk
Gaawaale jumtukaayi matt kode fu ñu mëna seentu, toppalante token yuy baamtu jur taxawaayu nangug projet bu yéeg
Risk yi ak balustrade yi
Optimize benn benchmark mën na nëbb ñakk kattan yu gëna yaatu ci sistem bi.
Njëg li ñuy fay ci infrastructure yi ak ci toppatoo dañuy faral di suufeel.
Bu sistem yi di gëna xawa jafee xam, jafe-jafe yi am ci wàllu kaaraange ak seetlu mën nañu gëna bari.
Roadmap ngir samp gi
Mandargal latency, kalite, ak njëg yi laata ngay jëfandikoo.
Benchmark ci biir sargal ak done yu dëggu.
Jumtukaay bi di saytu njuumte yi, derive bi ak njeextalu jëfandikukat bi.
Waajal rollback ak yooni tontu ci jafe-jafe yi laata ngay eskale.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding with EAGLE quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Dekodage spekulatif
Laaj yi ñuy faral di laaj
What is Speculative Decoding with EAGLE?
Decodage speculatif dafay gaawlu inference model làkk bu mag ci bàyyi model projet bu ndaw mu xalaat ay token yu bari ci kanam, model bu mag bi dafay xool ci benn yoon. EAGLE xeetu art la bu mujjee ci niveau fonctionnalité bi, du ci niveau token bi, di joxe 2-4x speedups te amul benn perte ci kalite bi.
Lan mooy xalaat bu mag bi ci ginaaw dekodage spekulatif?
Benn defarkat bu ndaw dafay xalaat ay jeton yu bari ci kanam, ba noppi modelu target bu mag bi daf leen di xool ñoom ñaar, nangu prefix bi gëna gudd te gëna jub.
Lan moo wuutale EAGLE ak xeeti dekodaas yu njëkk ya?
EAGLE dafay wax luy xew ci model biñ bëgga jëfandikoo, ba noppi jëfandikoowaat boppu LM bi, loolu mooy defar ay bordure yu gëna dëggu, moo gën xeetu token yi kese.
Lan moo tax ñu jàpp ni dekodage spekulatif amul benn perte?
Ndax xeetu target bi dafay saytu token bu nekk buñ defar ak distribution boppam, génne giñ nangu mooy li target bi mënoon defar kese.
Ban jafe-jafe ci EAGLE's man-man drafting la feedback token bi njëkk di jàppale ci saafara?
Kondisioneer ci token bi njëkk dafay wàññi jafe-jafe yi ci wax luy am ci màndarga yiñ nëbb, gëna baaxal dëggug projet bi.
Lan la EAGLE-2 yokk ci EAGLE bi njëkk?
EAGLE-2 dugal na garabu projet dinaamik bu xam contexte, yaatal car yu am njariñ ngir yokk limu nangu.