Ururimi AI

Amahugurwa menshi yo guhanura

Multi-token prediction trains a language model with targets beyond the immediately following token.

Umva iki gitaboAI-byavuzwe · ~3 min

Kuriyi page3 min soma
  1. Incamake
  2. Kwibira cyane
  3. Gushyira mu bikorwa Isi
  4. Ejo hazaza h'amahugurwa menshi yo guhanura
  5. Kubishyira mubikorwa
  6. Komeza Ubushakashatsi
  7. Ibibazo bikunze kubazwa

Incamake

Implementations differ: the cited Meta study uses parallel prediction heads, while DeepSeek-V3 describes sequential auxiliary modules.

Kwibira cyane

Ordinary next-token training predicts the token following a supplied prefix. Better and Faster Large Language Models via Multi-token Prediction extends the target horizon: several future tokens are predicted from each training position. Its architecture uses a shared transformer trunk and independent output heads for different offsets. These heads include transformer layers and use a shared unembedding matrix, so calling every additional head merely a lightweight linear layer is not the paper’s exact architecture. The training loss combines future-token cross-entropy terms. For a four-target design, the first head predicts the token at t+1 and the fourth predicts t+4 from the available prefix representation. Those targets do not mean that inference can accept four arbitrary guesses without checking them. The paper’s basic inference mode retains next-token prediction and discards the other heads. It also investigates greedy self-speculative decoding that proposes and verifies additional tokens. Its measured gains and speedups belong to particular code and language experiments, model sizes, batch sizes, and decoding procedures. DeepSeek-V3’s report presents another MTP implementation. It sequentially predicts additional tokens through auxiliary modules, preserving a causal chain across prediction depths rather than using the same independent parallel-head design. This distinction matters when describing how each module receives its inputs. Both works study additional prediction targets, but sharing an objective label does not make their architectures identical. Neither report establishes that MTP is a default ingredient in every frontier model or guarantees a fixed speedup for every deployment.

Gushyira mu bikorwa Isi

Comparing next-token and multi-token training under matched evaluation conditions.

Inspecting future-token targets at explicitly stated sequence offsets.

Using only the next-token head in the Meta study’s basic inference setup.

Studying verified self-speculative decoding with auxiliary predictions.

Ejo hazaza h'amahugurwa menshi yo guhanura

Experiments can vary prediction horizon, auxiliary-loss weight, architecture, and decoding strategy while measuring quality and resource cost. A deeper prediction horizon is a hypothesis to evaluate, not proof that a model plans correctly over longer tasks. Reports should separate improvements from additional supervision from gains due to a particular inference implementation. Coding benchmarks, text prediction, and algorithmic tasks can reveal different behavior. The useful question for a chosen system is whether its tested quality, memory, and latency justify the extra training or inference components under the intended workload.

Kubishyira mubikorwa

Impamvu ari ngombwa

  • Umuvuduko n'igipimo. Ururimi rwakazi rushobora kugenda byihuse nta gutamba guhuzagurika.
  • Kugera no kugera. Yagura uburyo bwindimi nuburyo bwo gutumanaho.
  • Ibyemezo bisobanutse. Amakipe arashobora kumara umwanya munini murubanza mugihe automatike ikora gusubiramo.

Witondere

  • Ibintu bifatika bishobora kwinjiza bucece raporo, gushyigikira imigendekere, cyangwa ibisubizo byubushakashatsi.
  • Kwihuta byihuse birashobora gukora ibisubizo bidahuye mubisabwa bisa.
  • Ibyanditswe byumvikana birashobora kugaragara niba kugenzura kugenzura ari ntege.

Intambwe ikurikira

  1. Sobanura imiterere isohoka, amajwi, hamwe nubuziranenge mbere yo gutangira.
  2. Ibisubizo byibanze hamwe nisoko yizewe igihe cyose ukuri kwingirakamaro.
  3. Komeza kugenzura abantu kugenzura ibisubizo byinshi.
  4. Kurikirana uburyo bwo kunanirwa no kongera imyitozo cyangwa akazi gahoraho.

Komeza Ubushakashatsi

Ibaruwa y'amakuru y'ubuntu

Ibuka amakuru ya buri munsi ya AI

Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.

Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.

Isuzume

Take the Multi-Token Prediction Training quiz

Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.

Tangira ikibazo

Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano

Ibibazo bikunze kubazwa

What is Multi-Token Prediction Training?

Multi-token prediction trains a language model with targets beyond the immediately following token. Implementations differ: the cited Meta study uses parallel prediction heads, while DeepSeek-V3 describes sequential auxiliary modules.

In the cited Meta study, what architecture predicts several future tokens from a training position?

The heads target different offsets from a shared prefix representation; this is the specified parallel-head design.

Niyihe moderi yakoresheje cyane intego yo guhanura intego yo kwitegura?

DeepSeek-V3 yemeye intego ya MTP yo kunoza imikorere yamakuru no gukora decoding yibitekerezo.

What additional supervision does multi-token training supply at a prefix position?

Several targets contribute prediction losses under the stated objective.

How are auxiliary predictions used in the Meta paper’s self-speculative inference experiment?

The reported experiment uses verified greedy self-speculation; arbitrary decoding policies need their own guarantees.

Bigenda bite kumutwe wabafasha nyuma yimyitozo niba udakeneye kwihuta?

Imitwe yinyongera irahitamo kubohereza; kubijugunya bikomeza icyitegererezo kingana nicyitegererezo gikurikira-token.