技術指南

序列模型中的教師強制

Teacher forcing is the standard way to train models that generate sequences one token at a time: during training, the model is fed the correct previous token rather than its own guess.

聆聽此指南AI 語音敘述 · ~7 min

本頁8 分鐘閱讀
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. Comparing teacher-forced and free-running decoding on a toy task
  5. 現實世界的實施
  6. 序列模型中教師強迫的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

Named by Williams and Zipser in 1989, it makes training fast and stable and lets Transformers train on whole sequences in parallel, but it creates a gap between training and generation that researchers still study.

重點摘要

  1. Teacher forcing feeds the true previous token, not the model's own prediction, when training a sequence model.
  2. Williams and Zipser named the technique in a 1989 Neural Computation paper on recurrent network learning.
  3. Teacher forcing lets Transformer decoders compute all token predictions in parallel using a shifted input and a causal mask.
  4. The gap between training on true prefixes and generating from the model's own outputs is commonly called exposure bias.
  5. Scheduled sampling raised MSCOCO captioning BLEU-4 from 28.8 to 30.6 by gradually replacing true tokens with model samples during training.

深入探討

Many AI systems produce output as a sequence: words in a translation, characters in a transcript, tokens in a chatbot reply. These autoregressive models generate one element at a time, and each new element depends on the ones already produced. Training such a model raises a simple question: when it is learning to predict the next token, what should it see as the previous tokens, the correct ones from the training data or the ones it would have produced itself? Teacher forcing is the answer almost everyone uses: feed the correct ones.

The term comes from Ronald Williams and David Zipser's 1989 paper 'A Learning Algorithm for Continually Running Fully Recurrent Neural Networks' in the journal Neural Computation. Describing a variant of their real-time recurrent learning algorithm, they wrote about replacing a unit's actual output with the teacher signal, the desired value, in later computation whenever such a value exists, and named this technique teacher forcing. They reported that on oscillation tasks only the teacher-forced version could solve the problems in general, because supplying the desired values helps control the network's state for the steps that follow.

Here is the mechanism in a translation example. Suppose the target sentence is 'the cat sat'. The decoder starts with a start-of-sentence symbol and must predict 'the'. Under teacher forcing, whatever it actually predicted, the next input is the true word 'the', and it must predict 'cat'. Then the true 'cat' is fed in, and it predicts 'sat'. At each position the loss compares the predicted probability distribution with the true next word, usually with cross-entropy, which rewards giving high probability to the correct token. Summed over positions, this is exactly maximum likelihood training of the sequence: the probability of a sentence factors into the product of each token's probability given the true tokens before it.

The benefits are concrete. Errors do not snowball during training, because one bad prediction never pollutes the inputs for later steps, so gradients carry clear signal and training converges faster, as the PyTorch translation tutorial notes. Teacher forcing also enables parallelism. In the Transformer, introduced in 'Attention Is All You Need' in June 2017, the decoder's inputs are the target sequence shifted right by one position, and a mask stops each position from attending to later ones. Because all the true previous tokens are known in advance, every position's prediction can be computed in one pass instead of one step at a time. This is a large part of why Transformers trained so efficiently; the paper reports 41.8 BLEU on WMT 2014 English-to-French after 3.5 days on eight GPUs. Modern language model pretraining is this same objective at scale.

The cost is a mismatch, often called exposure bias. At inference there is no teacher, so the model conditions on its own outputs, a distribution of prefixes it never saw during training. Samy Bengio and colleagues at Google described the problem in 2015: a mistake early in generation can put the model in a state rarely seen in training, and errors can accumulate. The PyTorch tutorial puts the symptom plainly: a teacher-forced network may show instability when run freely, and can produce grammatical output that drifts from the correct translation.

Several remedies followed. Scheduled sampling, by Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer (June 2015), flips a coin at each step during training to use either the true token or the model's own sample, and gradually shifts the odds from mostly true to mostly sampled with a linear, exponential or inverse sigmoid schedule. On MSCOCO image captioning, their Table 1 reports BLEU-4 rising from 28.8 for the baseline to 30.6 with scheduled sampling, while always feeding the model's own samples collapsed to 11.2. Sequence-level training by Marc'Aurelio Ranzato and colleagues (ICLR 2016) instead optimized the evaluation metric, such as BLEU, over whole generated sequences using reinforcement learning, mixed with teacher-forced training. Professor Forcing, by Alex Lamb, Anirudh Goyal and colleagues at NIPS 2016, used an adversarial discriminator to make the network's hidden-state dynamics look the same with and without teacher forcing.

The practical picture is mixed. Scheduled sampling's coin flips make each step depend on earlier sampled outputs, which removes the Transformer's parallel training advantage, and sequence-level reinforcement learning is noisier and costlier than maximum likelihood. As a result, plain teacher forcing remains the default for pretraining, with other methods layered on afterward, such as decoding strategies at inference or reinforcement learning fine-tuning. Research has also questioned the usual story. In 'The pitfalls of next-token prediction' (ICML 2024), Gregor Bachmann and Vaishnavh Nagarajan argue that compounding errors at inference are not the only problem: on certain planning tasks, teacher forcing can fail to learn an accurate next-token predictor at all, and they report that both Transformer and Mamba models fail on a minimal task of this kind.

In cost terms, teacher forcing is the cheap option. It requires only the training sequences themselves, one forward pass per sequence on parallel architectures, and a standard cross-entropy loss. Alternatives that expose the model to its own outputs require sequential sampling during training and often extra components, such as a reward model or a discriminator.

04工作範例

Comparing teacher-forced and free-running decoding on a toy task

  1. Build a small character-level recurrent model in PyTorch that learns to copy short strings, such as 'hello', followed by an end symbol.

  2. Train it with teacher forcing: at each step, feed the true previous character and compute cross-entropy against the true next character.

  3. After training, run it free: start from the start symbol and feed back the model's own most likely character each step.

  4. Record the per-character accuracy in both modes on a held-out set of strings, including some longer than any seen in training.

  5. Retrain with a scheduled-sampling rule that replaces the true character with the model's prediction with a probability that rises over training.

  6. Compare free-running accuracy for both models, especially on the longer strings where early mistakes have more steps to compound.

它顯示了什麼

The learner sees near-perfect accuracy for the teacher-forced model when given true prefixes, and a lower score when it runs on its own outputs, especially on long strings. The comparison shows exposure bias directly and whether scheduled sampling narrows the gap on this task.

現實世界的實施

Large language models are pretrained by next-token prediction with teacher forcing, the training setup that Bachmann and Nagarajan's 2024 paper on the pitfalls of next-token prediction analyzes in Transformer and Mamba models.

The PyTorch sequence-to-sequence translation tutorial uses teacher forcing in its decoder, noting that it speeds convergence but can make the trained network unstable when run on its own outputs.

Google researchers used scheduled sampling, a gradual move away from teacher forcing, in the system that ranked first in the 2015 MSCOCO image captioning challenge.

The original 2017 Transformer translation model trains its decoder on the target sentence shifted right by one position, with a mask hiding later words, so every target word is predicted from the true earlier words.

Professor Forcing, presented at NIPS 2016, trained recurrent models for handwriting, character-level text and raw-waveform speech so their behavior with and without teacher forcing looked alike.

序列模型中教師強迫的未來

Teacher forcing remains the core of language model pretraining, so its limits attract steady research attention. One line asks whether next-token prediction under teacher forcing can learn tasks that require planning ahead. Bachmann and Nagarajan (ICML 2024, revised in July 2025) give preliminary evidence that a teacherless objective, which uses dummy tokens so the model must predict several future tokens without seeing them, can avoid the failure they identify. Another line revisits the idea behind Ranzato and colleagues' sequence-level training: closing the train-inference gap by scoring and learning from the model's own complete outputs rather than only from true prefixes. Open questions include how much exposure bias actually harms large modern models compared with other error sources, and which training objectives best support long-horizon reasoning. Published results do not yet settle whether any alternative will replace teacher-forced next-token prediction as the default pretraining objective.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. Neural Computation (via ML Anthology)A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
  2. arXivScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
  3. arXiv (ICLR 2016)Sequence Level Training with Recurrent Neural Networks
  4. arXiv (NIPS 2016)Professor Forcing: A New Algorithm for Training Recurrent Networks
  5. arXiv (NeurIPS 2017)您所需要的就是關注
  6. PyTorchNLP From Scratch: Translation with a Sequence to Sequence Network and Attention
  7. arXiv (ICML 2024)The pitfalls of next-token prediction

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加序列模型中的教師強制測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是序列模型中的教師強迫?

Teacher forcing is the standard way to train models that generate sequences one token at a time: during training, the model is fed the correct previous token rather than its own guess. Named by Williams and Zipser in 1989, it makes training fast and stable and lets Transformers train on whole sequences in parallel, but it creates a gap between training and generation that researchers still study.

在教師強制期間,每個解碼步驟的輸入是什麼?

教師強制提供真實的先前目標標記而不是模型的預測,從而保持調節前綴正確。

在教師強制訓練期間,與解碼器輸入 y_(t−1) 配對的目標是什麼?

在教師強制中,解碼器接收 y_(t−1) 作為上下文,而交叉熵目標是真實的下一個 token y_t。

在 Transformer 解碼器中,什麼機制可以讓教師強制訓練跨位置並行運作?

因果掩碼可以防止每個位置專注於未來的標記,因此可以立即處理整個序列而不會作弊。

在推理時,為什麼不能使用教師強制?

在實際生成過程中,不存在目標序列,因此模型必須回饋自己的預測,這與訓練期間不同。

在教師強制訓練期間,每一步通常會使用哪一種損失?

自迴歸模型使用令牌級交叉熵進行訓練,將預測分佈與黃金下一個令牌進行比較。