Taggat ci wax luy am ci token yu bari
Duñu wax luy waaja am ci token bi ci topp, dañu tàggat model bi mu wax luy waaja am ci token yu bari ci ëlëg benn yoon.
Résumé
This sharpens learning signals and unlocks faster inference through self-speculative decoding.
Plongeur bu xóot
Modèlu làkk yiñ miin dañu leen di tàggat ci wax luy waaja am ci token bi ci topp: ñu jox leen benn contexte, wax luy waaja am ci token bi ci topp. Xalaatal token yu bari (MTP), siiwal ci benn këyit 2024 Meta te ñu jël ko ci DeepSeek-V3, yokk yeneen boppu génne yu woyof suko defee model bi di wax luy waaja am ci saasi token bi ci topp boole ci 2e, 3e token, ak 4e etaa same. Loolu dafay forse reso bi mu waajal ëlëg, ba noppi gëna dëgëral siñaalu tàggat bi - position bu nekk leegi dafay jàppale ay terme yu bari. Meta dafa wane njariñ yu bari ci wàllu kodage ak xalaat yuy jur, ak xeetu gëna mag yu gëna am njariñ. Li gëna am solo mooy, mën nañu sànni bopp yu bari yi ginaaw bi ñu tàggatee, kon dayo model bi ci deployment bi mënul màgg.
Gis-gis xarala
MTP dafay takk n boppu wax luy waaja am ci kaw bagaas transformatër biñ bokk; Bopp k dafay wax luy waaja am ci barab t+k ci misaal bi ci barab t. Perte yi ñu ngi leen boole ci diiru tàggat yaram. Ci inference, boppu auxiliaire yi dañuy may decodage self-speculatif: model bi dafay proposer ay token yu bari ci benn pass, ba noppi mu xool leen, dem ba yegg ci 3x generation bu gëna gaaw te du soppi distribution biy génn.
njeextalu pexe
Gaawaay ak yaatuwaay
Liggéeyukaay yi ci làkk yi mën nañu gëna gaaw te duñu yàq deggoo gi.
Dugg ak yegg
Dafay yaatal jëfandikoo gi ci làkk yi ak ci anam yi ñuy jokkoo.
dogal yu gëna leer
Ekip yi mën nañu gëna yàgg ci àtte ci jamono ji otomatisation di liggéey ci baamtu.
Ëlëgu tàggat yaram ci xam luy am ci ay token yu bari
MTP mingi nekk ingredient biñ gëna jëfandikoo ci rëset yiy tàggat frontiere ndax dafay gëna baaxal kalite bi ak gaawaay bi ci njëg yu néew. Xaarandi lëkkaloo bu gëna dëgër ak dekodaas speculatif, horizon yu gëna xóot ci wax luy am, ak jëfandikoo ko ni mébet buy jàppale ngir gëna mëna waajal horizon yu gudd yi. Buñu ko boole ak misaali xalaat, wax luy waaja am ci jéego yu bari mën na jàppale misaal yi ñu gëna xam njeexte yi balaa ñuy jël tontu.
Doxal ci àdduna dëgg
DeepSeek-V3 dafay jëfandikoo ab mébet MTP ci diiru tàggat ngir yokk njariñu done yi ak may dekodaas bu xalaat
AIU_PROTECTED_13_ modelu defar kode yi dañuy wane njariñu njub ci HumanEval ak MBPP ci xam luy waaja am ci token yu bari
Decodage speculatif sa bopp: defar 3-4 jetons ci paas bu nekk, ba noppi xool ngir gëna gaaw, distribution-conservation génne
Autocomplete bu gëna gaaw ci assistant codage fu ñuy proposé ay token yu bari te ñu xool ko ci benn jéego
Risk yi ak balustrade yi
Lépp lu jaarul yoon mën na dugg ci rapoor yi, jàppale ci liggéey bi, wala ci njariñu gëstu bi.
Sensibilite bu gaaw mën na jur njariñ yu wuute ci laajte yu noonu mel.
Done yu am solo mën nañu feeñ sudee seytu jëfandikoo gi néew doole.
Roadmap ngir samp gi
Mandargal formaa génne gi, melokaan bi, ak standard kalite yi laata ngay dugal ko.
Tontu yu am solo ak balluwaay yu wóor saa yu dëggu bi di am solo.
Fexeel am barabu xool nit ñi ngir am njariñ yu am solo.
Toppal anami gacce yi ak di faral di tàggataat ay laaj wala def-liggéey.
Weyal di banneexu
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Multi-Token Prediction Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gis bi ci topp
Streaming buy xalaat ak wax luy am ci token yu bari
Laaj yi ñuy faral di laaj
What is Multi-Token Prediction Training?
Duñu wax luy waaja am ci token bi ci topp, dañu tàggat model bi mu wax luy waaja am ci token yu bari ci ëlëg benn yoon. Loolu dafay ñawloo siñaal yi ñuy jàng, ba noppi ubbi inference bu gëna gaaw ci decodage speculatif.
Lan la prediction multi-token di yokk suñu ko méngale ak tàggat-token bi ci topp?
MTP dafay yokk yeneen boppu génne suko defee model bi mëna wax luy waaja am ci token bi ci topp boole ci yeneen token yu gëna sori ci benn réew bu nëbbu.
Ban model moo gëna jëfandikoo ab mébetu wax luy waaja am ci pretraining?
DeepSeek-V3 jël na mébetu tàggat MTP ngir gëna baaxal done yi ak mëna dekodage spekulatif.
Lu tax MTP di gëna dëgëral siñaalu tàggat bi?
Xalaatal ay token yu bari ci ëlëg dafay tekki ni position token bu nekk dafay defar ay perte yu bari ci ëlëg, di joxe siñaal jàng bu gëna bari ci token bu nekk.
Ban njariñu inference la boppu prediction yu bees yi mëna am?
Bopp yiy jàppale mën nañu defar ay jeton yu bari ci paas bu nekk ñu xool ko bu baax, gaawlu defar bi te du soppi distribution biy génn.
Lan mooy dall boppu ndimbal yi ginaaw tàggat yaram sudee soxlawolen gaawaay?
Bopp yu bees yi mën nañu leen tànn ci deployment bi; sànni leen dafay tax model bi nekk benn dayo ak model token bi ci topp.