HejuruUbuyobozi bukurikira
Gutondeka gutemba no guhanura byinshi
Tekiniki
Ururimi AI
Multi-token prediction trains a language model with targets beyond the immediately following token.
Umva iki gitaboAI-byavuzwe · ~3 min
Implementations differ: the cited Meta study uses parallel prediction heads, while DeepSeek-V3 describes sequential auxiliary modules.
Ordinary next-token training predicts the token following a supplied prefix. Better and Faster Large Language Models via Multi-token Prediction extends the target horizon: several future tokens are predicted from each training position. Its architecture uses a shared transformer trunk and independent output heads for different offsets. These heads include transformer layers and use a shared unembedding matrix, so calling every additional head merely a lightweight linear layer is not the paper’s exact architecture. The training loss combines future-token cross-entropy terms. For a four-target design, the first head predicts the token at t+1 and the fourth predicts t+4 from the available prefix representation. Those targets do not mean that inference can accept four arbitrary guesses without checking them. The paper’s basic inference mode retains next-token prediction and discards the other heads. It also investigates greedy self-speculative decoding that proposes and verifies additional tokens. Its measured gains and speedups belong to particular code and language experiments, model sizes, batch sizes, and decoding procedures. DeepSeek-V3’s report presents another MTP implementation. It sequentially predicts additional tokens through auxiliary modules, preserving a causal chain across prediction depths rather than using the same independent parallel-head design. This distinction matters when describing how each module receives its inputs. Both works study additional prediction targets, but sharing an objective label does not make their architectures identical. Neither report establishes that MTP is a default ingredient in every frontier model or guarantees a fixed speedup for every deployment.
Comparing next-token and multi-token training under matched evaluation conditions.
Inspecting future-token targets at explicitly stated sequence offsets.
Using only the next-token head in the Meta study’s basic inference setup.
Studying verified self-speculative decoding with auxiliary predictions.
Experiments can vary prediction horizon, auxiliary-loss weight, architecture, and decoding strategy while measuring quality and resource cost. A deeper prediction horizon is a hypothesis to evaluate, not proof that a model plans correctly over longer tasks. Reports should separate improvements from additional supervision from gains due to a particular inference implementation. Coding benchmarks, text prediction, and algorithmic tasks can reveal different behavior. The useful question for a chosen system is whether its tested quality, memory, and latency justify the extra training or inference components under the intended workload.
Ibaruwa y'amakuru y'ubuntu
Inkuru eshatu zizewe za AI buri gitondo cy'iminsi y'akazi, zanditse mu rurimi rworoshye. Ubuntu iteka ryose, nta kwamamaza.
Imeri imwe buri munsi y'akazi. Kwiyimura mu buryo bumwe. Ntitwigera tugurisha cyangwa ngo dusangire aderesi yawe.
Isuzume
Ibisubizo ako kanya kuri buri kibazo, n'icyemezo gishobora gusangirwa gifite ID yagenzurwa umaze gutsinda isomo.
Fasha uburezi bwa AI bw'ubuntu. AI Understanding ni umuryango utegamiye kuri leta wa 501(c)(3) — nta kwamamaza, nta kwishyura, iteka ryose. Tanga impano
Multi-token prediction trains a language model with targets beyond the immediately following token. Implementations differ: the cited Meta study uses parallel prediction heads, while DeepSeek-V3 describes sequential auxiliary modules.
The heads target different offsets from a shared prefix representation; this is the specified parallel-head design.
DeepSeek-V3 yemeye intego ya MTP yo kunoza imikorere yamakuru no gukora decoding yibitekerezo.
Several targets contribute prediction losses under the stated objective.
The reported experiment uses verified greedy self-speculation; arbitrary decoding policies need their own guarantees.
Imitwe yinyongera irahitamo kubohereza; kubijugunya bikomeza icyitegererezo kingana nicyitegererezo gikurikira-token.
Komeza wige
Abandi bayobozi batoranijwe kuriyi ngingo
HejuruUbuyobozi bukurikira
Gutondeka gutemba no guhanura byinshi
Tekiniki