Multi-Token Prediction Training
Panzvimbo pekufanotaura chete chiratidzo chinotevera, modhi inodzidziswa kufanotaura akawanda emangwana tokeni kamwechete.
Pfupiso
This sharpens learning signals and unlocks faster inference through self-speculative decoding.
Kudzika Kwakadzika
Mienzaniso yemitauro yakajairwa inodzidziswa nekufanotaura kunotevera-chiratidzo: kupihwa mamiriro, fungidzira chiratidzo chinotevera. Multi-token prediction (MTP), yakafarirwa nebepa ra2024 Meta uye yakagamuchirwa muDeepSeek-V3, inowedzera mimwe misoro yakareruka yekubuda kuitira kuti modhi yacho ifanofembera panguva imwe chete chiratidzo chinotevera pamwe neyechipiri, 3, uye 4th tokens kumberi kubva munzvimbo imwechete yakavanzika. Izvi zvinomanikidza network kuronga mberi mune ramangwana uye inosimbisa chiratidzo chekudzidzisa - imwe neimwe nzvimbo ikozvino inopa akawanda kurasikirwa mazwi. Meta yakashuma zvakawandirwa zvakanyanya pakunyora kodhi uye kufunga kunovaka, nemhando dzakakura dzichibatsira zvakanyanya. Zvine hutsinye, iyo misoro yekuwedzera inogona kuraswa mushure mekudzidziswa, saka saizi yemhando pakutumirwa haifanire kukura.
Technical Insight
MTP inonamatira n yakazvimiririra yekufanotaura misoro pamusoro peiyo yakagovaniswa transformer trunk; musoro k unofanotaura chiratidzo pachinzvimbo t+k kubva pane chinomiririra pachinzvimbo t. Kurasikirwa kunopfupikiswa panguva yekudzidziswa. Pakunongedza, misoro yebetsero inogonesa kuzvifungira decoding: modhi inokurudzira tokeni akati wandei mukupasa kumwe chete, wobva wazvisimbisa, uchiwana kusvika kunosvika 3x inokurumidza chizvarwa pasina kushandura kugovera.
Strategic Impact
Kumhanya uye chiyero
Mutauro workflows inogona kufamba nekukurumidza pasina kupira kuenderana.
Svika uye svika
Inopamhidzira kupinda mumitauro yese nemataera ekutaurirana.
Sarudzo dzakajeka
Zvikwata zvinogona kupedza nguva yakawanda pakutonga uku otomatiki ichibata kudzokorora.
Ramangwana reMulti-Token Prediction Training
MTP yave kuita chigadziriso mumiganho yekudzidzira mabikirwo nekuti inovandudza ese emhando uye inference kumhanya pamutengo mudiki. Tarisira kubatanidzwa kwakasimba nekufungidzira decoding, zvakadzama kufanotaura horizons, uye shandisa sechinangwa chekubatsira chinovandudza kuronga kwenguva refu. Yakasanganiswa nemhando dzekufunga, kufanotaura nhanho dzakawanda kumberi kunogona kubatsira modhi mukati kutevedzera mhedzisiro isati yazvipira kumhinduro.
Real-World Implementation
DeepSeek-V3 uchishandisa chinangwa cheMTP panguva yekudzidzira kuwedzera kushanda kwedata uye kugonesa kufungidzira decoding.
Meta's macode-generation modhi anoratidza huchokwadi hunowanikwa paHumanEval neMBPP kubva mukufanotaura tokeni dzakawanda.
Kuzvifungira wega decoding: kunyora 3-4 tokens per forward pass wobva waona kukurumidza, kugovera-kuchengetedza kubuda.
Inokurumidza kuzadzisa otomatiki mumakodha vabatsiri uko akawanda anonzwisisika tokeni anotsanangurwa uye anotariswa mune imwe nhanho.
Njodzi & Guardrails
Chokwadi chehuroyi chinogona kupinda chinyararire mishumo, kuyerera kwetsigiro, kana tsvakiridzo.
Kunzwa nekukasira kunogona kugadzira mhedzisiro isingaenderane pane zvikumbiro zvakafanana.
Sensitive text data inogona kuburitswa kana zvidhiraivho zvisina kusimba.
Implementation Roadmap
Tsanangura chimiro chekubuda, toni, uye mhando zviyero usati waburitsa.
Mhinduro dzepasi neakavimbika masosi pese pazvine basa.
Chengetedza ongororo yekuongorora yemunhu kune yakakwira-stake zvinobuda.
Tevera maitiro ekutadza uye dzidzisazve kukurudzira kana mafambiro ebasa nguva nenguva.
Ramba Uchiongorora
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Multi-Token Prediction Training quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Gaidhi rinotevera
Kufungidzira Kutenderera uye Multi-Token Prediction
Mibvunzo inowanzo bvunzwa
What is Multi-Token Prediction Training?
Panzvimbo pekufanotaura chete chiratidzo chinotevera, modhi inodzidziswa kufanotaura akawanda emangwana tokeni kamwechete. Izvi zvinorodza masaini ekudzidza uye kuvhura kukurumidza kufungidzira kuburikidza nekuzvifungira decoding.
Chii chinowedzera fungidziro yezviratidzo zvakawanda zvichienzaniswa neyakajairwa-tokeni kudzidziswa?
MTP inowedzera mimwe misoro inobuda kuitira kuti modhi inofanotaura tokeni inotevera pamwe nematinji akati wandei kumberi kubva kune imwe yakavanzika nyika.
Ndeipi modhi yakanyatso shandisa yakawanda-token kufanotaura chinangwa mu pretraining?
DeepSeek-V3 yakatora chinangwa cheMTP chekudzidzisa kuvandudza kugona kwedata uye kugonesa kufungidzira decoding.
Sei MTP ichisimbisa chiratidzo chekudzidzisa?
Kufanotaura akati wandei emangwana tokeni kunoreva kuti imwe neimwe nzvimbo yechiratidzo inoburitsa kurasikirwa kwakati wandei, ichipa yakawanda yekudzidza chiratidzo pachiratidzo.
Ndedzipi fungidziro inobatsira iyo mimwe misoro yekufungidzira inogonesa?
Misoro yekubatsira inogona kunyora akawanda tokeni papasi rinobva rasimbiswa, ichimhanyisa chizvarwa pasina kushandura kugovera kwekubuda.
Chii chinoitika kumisoro yekubatsira mushure mekudzidziswa kana iwe usingade kukurumidza-ups?
Misoro yekuwedzera inosarudzika pakutumirwa; achivarasa anochengeta modhi saizi yakafanana neyakajairwa inotevera-chiratidzo modhi.