マルチトークン予測トレーニング

Ordinary next-token training predicts the token following a supplied prefix. Better and Faster Large Language Models via Multi-token Prediction extends the target horizon: several future tokens are predicted from each training position. Its architecture uses a shared transformer trunk and independent output heads for different offsets. These heads include transformer layers and use a shared unembedding matrix, so calling every additional head merely a lightweight linear layer is not the paper’s exact architecture. The training loss combines future-token cross-entropy terms. For a four-target design, the first head predicts the token at t+1 and the fourth predicts t+4 from the available prefix representation. Those targets do not mean that inference can accept four arbitrary guesses without checking them. The paper’s basic inference mode retains next-token prediction and discards the other heads. It also investigates greedy self-speculative decoding that proposes and verifies additional tokens. Its measured gains and speedups belong to particular code and language experiments, model sizes, batch sizes, and decoding procedures. DeepSeek-V3’s report presents another MTP implementation. It sequentially predicts additional tokens through auxiliary modules, preserving a causal chain across prediction depths rather than using the same independent parallel-head design. This distinction matters when describing how each module receives its inputs. Both works study additional prediction targets, but sharing an objective label does not make their architectures identical. Neither report establishes that MTP is a default ingredient in every frontier model or guarantees a fixed speedup for every deployment.