Decodage parallèle yax-de-pensée
Skeleton-of-Thought (SoT) xarala la buy ñaax ak dekode, muy njëkka laaj modelu làkk bi mu tëral ab tontu bu gàtt, ba noppi mu yaatal tontu bu nekk ci paralel.
Résumé
It matters because it can cut the wall-clock latency of long answers by roughly 2x without retraining the model.
Plongeur bu xóot
Modèlu làkk yu mag yi dañuy defar benn token benn yoon, kon tontu bu gudd dafay yeex ndax baat bu nekk dafay xaar ki ko jiitu. Yaxu xalaat, bi gëstukat yi dugal ci Tsinghua ak Microsoft ci 2023, dafay yamale liggéey bi. Woo bu njëkk bi dafay laaj model bi benn yax bu gàtt: limu 3 ba 10 poñ yu am nimero, bu nekk ci ñoom ay baat yu néew. Ñaareelu mbooloo mu woote dafay yaatal poñ bu nekk ci boppam ak ci benn yoon, ndax poñ yi bokku ñu ci seen biir. Expansion yi dañu leen boole ci tontu bu mujj bi. Ndax etape expansion bu yeex bi dafay dox ci paralel, latency total bi dafay wàññeeku bu baax ci laaj yi tontu yi di xaajaloo ci pàcc yu moom seen bopp, lu melni lim ay xelal wala méngale tànneef yi.
Gis-gis xarala
SoT dafay jàpp ni decoder inference bi dafay tënku ci latency, du saa yu nekk lay tënk ci xayma: benn laaj dafay faral di bàyyi GPU bi ñu jëfandikoo ko bu baax. Doxal point expansions ni batch dafay tax hardware bi nekk busy te dafay jaxasoo ci generation point bu nekk. Ak xeetu API yi, dañuy joxe yokkute yi ci laaj yuy boole; ak xeetu dëkk bi, dañu bokk benn jéego bu jëm kanam. Etape skeleton bi dafay yokk benn bopp bu gàtt, kon gaawaayu net bi dafay màgg ak guddaayu tontu bi ak limu poñ yi moom seen bopp.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu dekodage paralel bu yax-ci-xalaat
Xaarandi xalaati SoT ngir boole ci yoon wiñ mëna ànd: sistem yi dina ñu gis su laajte bi settee ba noppi soppi ci yaatal paralel, daanu ci xalaat yu toppalante ngir liggéey yu jëm ci liggéey yu melni firnde math. Variante yu melni SoT ak dependence graph dynamique dañuy may poñ yi ñuy royuwaale seen biir. Ginaaw kaadar yiy liggéey dañuy yokk ndimmbalu laajte yu ndaw ak dekodaas speculatif, pexe dekodaas paralel dina nekk layer buy wàññi latency buñ miin, du nekk luñuy def ci loxo.
Doxal ci àdduna dëgg
Gaawaay chatbot buy tontu 'jox ma 8 xalaat ngir wàññi njëgu cloud' ci yaatal juróom ñatti xalaat yi benn yoon.
Jàppalekat biy jàppale kiliyaan bi di defar téere buy saafara jafe-jafe ci wàll yu bari te gëna néew tontu.
Defar ab tontu buy méngale (njariñ ak loraange ñaari produit) fu bullet bu nekk di fees benn yoon.
Sistem yiy liggéey ci backend dañuy boole wàll tontu yu moom seen bopp ngir yokk jëfandikoo GPU ci diiru defar formileer yu yàgg yi.
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Skeleton-of-Thought Parallel Decoding quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Dekodage jeton paralel MaskGIT
Laaj yi ñuy faral di laaj
What is Skeleton-of-Thought Parallel Decoding?
Skeleton-of-Thought (SoT) xarala la buy ñaax ak dekode, muy njëkka laaj modelu làkk bi mu tëral ab tontu bu gàtt, ba noppi mu yaatal tontu bu nekk ci paralel. Dafa am solo ndax mën na wàññi latency wall-clock ci tontu yu gudd yi ci lu tollu ci 2x te doo tàggataat model bi.
Lan la etap bu njëkk ci Yax-ci-Xalaat di jur?
SoT dafay njëkka wax model bi mu génne benn skeleton bu gàtt bu am boppu poñ, ñu daal di koy yaatal bu nekk.
Lan moo waral etapu yaatuwaayu poñ yi mëna dox paralel?
Ñu ngi defar point skeleton yi ñu mëna moom seen bopp, kon yaatal leen jarul nga xaar ay mbokk.
Lan mooy jafe-jafe bi gëna mag ci SoT ci dekodage LLM bu jaar yoon?
Decodage buñ miin dafay tënku ci latency ndax token bu nekk dafay xaar ki ko jiitu; SoT dafay jaxasoo liggéey ngir wàññi waxtu montor bi.
Ci ban xeetu laaj la SoT di joxe gaawaay bu gëna mag?
Tontu yiy xaajaloo ci pàcc yu bari yu moom seen bopp ñooy gëna am njariñ, ndax pàcc bu nekk dafay yaatu ci jamono jooju.
Ban njëg la SoT di yokk suñu ko méngale ak benn xeetu liggéey bu toppalante?
Etape skeleton bi dafay yokk ab latency bu ndaw bu takku, buñu ëppee doole ci yaatuwaayu paralel ci tontu yu gudd yi.