Modèlu projet de decodage speculatif
Decodage speculatif dafay jëfandikoo model 'projet' bu ndaw te gaaw ngir xayma ay token yu bari yuy ñëw yu model bu mag di xool ci benn yoon.
Résumé
It speeds up text generation 2-3x with no change to the output.
Plongeur bu xóot
Modèlu làkk yu mag yi dañuy defar mbind benn token benn yoon, te jéego bu nekk dafay laaj jàll bu mat sëkk ci ay miliyaar ciy parametre — yu yeex te tëju ci mémoire. Decodage speculatif dafay song lii ci boole xeetu 'target' bu rëy bi ak xeetu 'projet' bu yomb bi. Xeetu projet bi dafay gaaw di proposer benn pàcc, nu wax, 4-8 jetons kàndidaa. Modèle bu mag bi dafay liggéey ñoom ñépp ci benn pass forward parallèle ba noppi xool bu nekk ci ñoom. Token yi méngoo ak li model bu mag bi di defar, dañu leen di nangu; njuumte bu njëkk bi dañu koy saafara ba noppi sànni leneen li. Ndax xool ay jeton yu bari ci benn yoon lu tollu ci benn njëg ak defar benn, daw yi ñuy nangu daanaka duñu fay. Lu am solo mooy, jéego buy bañ-échantillonnage dafay garanti séddale bu mujj bi dafay nuru doxal model bu mag bi kese - gaawaay bu amul benn perte ci kalite.
Gis-gis xarala
Li gëna am solo mooy ñu soppi test biy jël misaal ngir bañ ko. Bépp jeton buñ bind, dañuy méngale probabilite model bi ñuy wut ak model biñ defar. Sudee mébet bi dafa joxe probabilite bu tolloo wala bu gëna rëy, dañuy nangu jeton bi; luko moy dañu koy nangu ak probabilite bu méngoo ak ratio bi, te suñu ko bañee dañuy jël token buñ jubbanti ci distribution residuelle buñ yamale. Math bii dafay wane ni génne gi dafay méngoo ak sampling bi juge ci model bu mag bi.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu modelu projet de decodage spekulatif
Xaarandil ni xeetu projet yi dina ñu nekk jumtukaay yuñ miin ci serveur yu melni vLLM ak TensorRT-LLM. Self-speculation variants (Medusa, EAGLE) dañu bàyyi xeetu projet bu wuute bi yépp ci yokk boppu prediction yu woyof, ak projet bu sukkandiko ci garab dafay firndeel wéyal kàndidaa yu bari benn yoon. Lu palanteer yi di gëna màgg, njëgu liggéey bi di gëna am doole, ñi gëna am xel, ñi méngoo ak model yi ak xool bu baax ci hardware bi dina gëna yokk limu nangu yi ak génne gi.
Doxal ci àdduna dëgg
Anthropic, OpenAI, and Google use speculative decoding to cut latency and serving cost on chat assistants serving millions of users.
vLLM ak NVIDIA TensorRT-LLM dañuy indi dekodaas buy xalaat suko defee ñiy jëfandikoo seen bopp mëna gaaw ci dugal Llama wala Mistral.
Teg xeetu projet 7B ak 70B target (lu melni, famiy Llama-3) ngir lu tollu ci ñaari yoon jetons-per-second ci benn GPU.
Jumtukaayi matt kode yi dañuy jëfandikoo ab modelu projet bu ndaw ngir proposer boilerplate bi model bi gëna mag di xool, ba noppi tëye xalaat yi ci editër bi.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Speculative Decoding Draft Models quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Coppite yu am solo ngir modeli kode
Laaj yi ñuy faral di laaj
What is Speculative Decoding Draft Models?
Decodage speculatif dafay jëfandikoo model 'projet' bu ndaw te gaaw ngir xayma ay token yu bari yuy ñëw yu model bu mag di xool ci benn yoon. Dafay gaaw defar mbind 2-3x te amul benn coppite ci génne gi.
Lan mooy cër bu njëkk bi ci xeetu 'projet' ci dekodage spekulatif?
Modèle projet bu ndaw bi dafay xayma jeton yiy ñëw ci anam wu yomb, te modelu target bu mag bi dafay xool ci benn pass parallèle.
Lan moo waral xool token yu bari yuñ bind benn yoon dina sakkanal jot?
Generation mingi aju ci fàttaliku; xool ay jeton yu bari ci benn pass batched daanaka yomb na benn jéego, kon daw yi ñuy nangu daanaka duñu fay.
Lan mooy dall token yiñ bind ginaaw token bi njëkk bi model biñ bëgga bañ?
Nangu gi dafay wéy ba keroog ñu njëkka ñàkka deggoo; token boobu dañu koy seet ba noppi sànni token yiñ ci topp yépp.
naka lay def ba decodage speculatif mëna wóor ni kalite biy génne du yàqu?
Test buñ soppeeku ngir jël misaal, dafay garanti ni distribution token bu mujj bi méngoo ak misaal biñ jëlee ci model biñ bëgga jël.
Ci yii, yan ci ñoom mooy 'self-speculation' biy moytu ab xeetu projet bu wuute?
Medusa ak EAGLE yokk nañu boppu prediction yu woyof ci model bu mag bi suko defee mu mëna defar ay token ëlëg.