Forțarea profesorului în modelele de secvență

Many AI systems produce output as a sequence: words in a translation, characters in a transcript, tokens in a chatbot reply. These autoregressive models generate one element at a time, and each new element depends on the ones already produced. Training such a model raises a simple question: when it is learning to predict the next token, what should it see as the previous tokens, the correct ones from the training data or the ones it would have produced itself? Teacher forcing is the answer almost everyone uses: feed the correct ones.

The term comes from Ronald Williams and David Zipser's 1989 paper 'A Learning Algorithm for Continually Running Fully Recurrent Neural Networks' in the journal Neural Computation. Describing a variant of their real-time recurrent learning algorithm, they wrote about replacing a unit's actual output with the teacher signal, the desired value, in later computation whenever such a value exists, and named this technique teacher forcing. They reported that on oscillation tasks only the teacher-forced version could solve the problems in general, because supplying the desired values helps control the network's state for the steps that follow.

Here is the mechanism in a translation example. Suppose the target sentence is 'the cat sat'. The decoder starts with a start-of-sentence symbol and must predict 'the'. Under teacher forcing, whatever it actually predicted, the next input is the true word 'the', and it must predict 'cat'. Then the true 'cat' is fed in, and it predicts 'sat'. At each position the loss compares the predicted probability distribution with the true next word, usually with cross-entropy, which rewards giving high probability to the correct token. Summed over positions, this is exactly maximum likelihood training of the sequence: the probability of a sentence factors into the product of each token's probability given the true tokens before it.

The benefits are concrete. Errors do not snowball during training, because one bad prediction never pollutes the inputs for later steps, so gradients carry clear signal and training converges faster, as the PyTorch translation tutorial notes. Teacher forcing also enables parallelism. In the Transformer, introduced in 'Attention Is All You Need' in June 2017, the decoder's inputs are the target sequence shifted right by one position, and a mask stops each position from attending to later ones. Because all the true previous tokens are known in advance, every position's prediction can be computed in one pass instead of one step at a time. This is a large part of why Transformers trained so efficiently; the paper reports 41.8 BLEU on WMT 2014 English-to-French after 3.5 days on eight GPUs. Modern language model pretraining is this same objective at scale.

The cost is a mismatch, often called exposure bias. At inference there is no teacher, so the model conditions on its own outputs, a distribution of prefixes it never saw during training. Samy Bengio and colleagues at Google described the problem in 2015: a mistake early in generation can put the model in a state rarely seen in training, and errors can accumulate. The PyTorch tutorial puts the symptom plainly: a teacher-forced network may show instability when run freely, and can produce grammatical output that drifts from the correct translation.

Several remedies followed. Scheduled sampling, by Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer (June 2015), flips a coin at each step during training to use either the true token or the model's own sample, and gradually shifts the odds from mostly true to mostly sampled with a linear, exponential or inverse sigmoid schedule. On MSCOCO image captioning, their Table 1 reports BLEU-4 rising from 28.8 for the baseline to 30.6 with scheduled sampling, while always feeding the model's own samples collapsed to 11.2. Sequence-level training by Marc'Aurelio Ranzato and colleagues (ICLR 2016) instead optimized the evaluation metric, such as BLEU, over whole generated sequences using reinforcement learning, mixed with teacher-forced training. Professor Forcing, by Alex Lamb, Anirudh Goyal and colleagues at NIPS 2016, used an adversarial discriminator to make the network's hidden-state dynamics look the same with and without teacher forcing.

The practical picture is mixed. Scheduled sampling's coin flips make each step depend on earlier sampled outputs, which removes the Transformer's parallel training advantage, and sequence-level reinforcement learning is noisier and costlier than maximum likelihood. As a result, plain teacher forcing remains the default for pretraining, with other methods layered on afterward, such as decoding strategies at inference or reinforcement learning fine-tuning. Research has also questioned the usual story. In 'The pitfalls of next-token prediction' (ICML 2024), Gregor Bachmann and Vaishnavh Nagarajan argue that compounding errors at inference are not the only problem: on certain planning tasks, teacher forcing can fail to learn an accurate next-token predictor at all, and they report that both Transformer and Mamba models fail on a minimal task of this kind.

In cost terms, teacher forcing is the cheap option. It requires only the training sequences themselves, one forward pass per sequence on parallel architectures, and a standard cross-entropy loss. Alternatives that expose the model to its own outputs require sequential sampling during training and often extra components, such as a reward model or a discriminator.