Műszaki ÚTMUTATÓ

Tanári kényszerítés a sorozatmodellekben

Teacher forcing is the standard way to train models that generate sequences one token at a time: during training, the model is fed the correct previous token rather than its own guess.

Hallgassa meg ezt az útmutatótAI-narrált · ~7 min

Ezen az oldalon8 perc olvasás
  1. Áttekintés
  2. Főbb tanulságok
  3. Mély merülés
  4. Comparing teacher-forced and free-running decoding on a toy task
  5. Valós megvalósítás
  6. A tanárkényszer jövője a sorozatmodellekben
  7. A gyakorlatba ültetés
  8. Források és további olvasmányok
  9. Folytassa a felfedezést
  10. Gyakran ismételt kérdések

Áttekintés

Named by Williams and Zipser in 1989, it makes training fast and stable and lets Transformers train on whole sequences in parallel, but it creates a gap between training and generation that researchers still study.

Főbb tanulságok

  1. Teacher forcing feeds the true previous token, not the model's own prediction, when training a sequence model.
  2. Williams and Zipser named the technique in a 1989 Neural Computation paper on recurrent network learning.
  3. Teacher forcing lets Transformer decoders compute all token predictions in parallel using a shifted input and a causal mask.
  4. The gap between training on true prefixes and generating from the model's own outputs is commonly called exposure bias.
  5. Scheduled sampling raised MSCOCO captioning BLEU-4 from 28.8 to 30.6 by gradually replacing true tokens with model samples during training.

Mély merülés

Many AI systems produce output as a sequence: words in a translation, characters in a transcript, tokens in a chatbot reply. These autoregressive models generate one element at a time, and each new element depends on the ones already produced. Training such a model raises a simple question: when it is learning to predict the next token, what should it see as the previous tokens, the correct ones from the training data or the ones it would have produced itself? Teacher forcing is the answer almost everyone uses: feed the correct ones.

The term comes from Ronald Williams and David Zipser's 1989 paper 'A Learning Algorithm for Continually Running Fully Recurrent Neural Networks' in the journal Neural Computation. Describing a variant of their real-time recurrent learning algorithm, they wrote about replacing a unit's actual output with the teacher signal, the desired value, in later computation whenever such a value exists, and named this technique teacher forcing. They reported that on oscillation tasks only the teacher-forced version could solve the problems in general, because supplying the desired values helps control the network's state for the steps that follow.

Here is the mechanism in a translation example. Suppose the target sentence is 'the cat sat'. The decoder starts with a start-of-sentence symbol and must predict 'the'. Under teacher forcing, whatever it actually predicted, the next input is the true word 'the', and it must predict 'cat'. Then the true 'cat' is fed in, and it predicts 'sat'. At each position the loss compares the predicted probability distribution with the true next word, usually with cross-entropy, which rewards giving high probability to the correct token. Summed over positions, this is exactly maximum likelihood training of the sequence: the probability of a sentence factors into the product of each token's probability given the true tokens before it.

The benefits are concrete. Errors do not snowball during training, because one bad prediction never pollutes the inputs for later steps, so gradients carry clear signal and training converges faster, as the PyTorch translation tutorial notes. Teacher forcing also enables parallelism. In the Transformer, introduced in 'Attention Is All You Need' in June 2017, the decoder's inputs are the target sequence shifted right by one position, and a mask stops each position from attending to later ones. Because all the true previous tokens are known in advance, every position's prediction can be computed in one pass instead of one step at a time. This is a large part of why Transformers trained so efficiently; the paper reports 41.8 BLEU on WMT 2014 English-to-French after 3.5 days on eight GPUs. Modern language model pretraining is this same objective at scale.

The cost is a mismatch, often called exposure bias. At inference there is no teacher, so the model conditions on its own outputs, a distribution of prefixes it never saw during training. Samy Bengio and colleagues at Google described the problem in 2015: a mistake early in generation can put the model in a state rarely seen in training, and errors can accumulate. The PyTorch tutorial puts the symptom plainly: a teacher-forced network may show instability when run freely, and can produce grammatical output that drifts from the correct translation.

Several remedies followed. Scheduled sampling, by Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer (June 2015), flips a coin at each step during training to use either the true token or the model's own sample, and gradually shifts the odds from mostly true to mostly sampled with a linear, exponential or inverse sigmoid schedule. On MSCOCO image captioning, their Table 1 reports BLEU-4 rising from 28.8 for the baseline to 30.6 with scheduled sampling, while always feeding the model's own samples collapsed to 11.2. Sequence-level training by Marc'Aurelio Ranzato and colleagues (ICLR 2016) instead optimized the evaluation metric, such as BLEU, over whole generated sequences using reinforcement learning, mixed with teacher-forced training. Professor Forcing, by Alex Lamb, Anirudh Goyal and colleagues at NIPS 2016, used an adversarial discriminator to make the network's hidden-state dynamics look the same with and without teacher forcing.

The practical picture is mixed. Scheduled sampling's coin flips make each step depend on earlier sampled outputs, which removes the Transformer's parallel training advantage, and sequence-level reinforcement learning is noisier and costlier than maximum likelihood. As a result, plain teacher forcing remains the default for pretraining, with other methods layered on afterward, such as decoding strategies at inference or reinforcement learning fine-tuning. Research has also questioned the usual story. In 'The pitfalls of next-token prediction' (ICML 2024), Gregor Bachmann and Vaishnavh Nagarajan argue that compounding errors at inference are not the only problem: on certain planning tasks, teacher forcing can fail to learn an accurate next-token predictor at all, and they report that both Transformer and Mamba models fail on a minimal task of this kind.

In cost terms, teacher forcing is the cheap option. It requires only the training sequences themselves, one forward pass per sequence on parallel architectures, and a standard cross-entropy loss. Alternatives that expose the model to its own outputs require sequential sampling during training and often extra components, such as a reward model or a discriminator.

04Működő példa

Comparing teacher-forced and free-running decoding on a toy task

  1. Build a small character-level recurrent model in PyTorch that learns to copy short strings, such as 'hello', followed by an end symbol.

  2. Train it with teacher forcing: at each step, feed the true previous character and compute cross-entropy against the true next character.

  3. After training, run it free: start from the start symbol and feed back the model's own most likely character each step.

  4. Record the per-character accuracy in both modes on a held-out set of strings, including some longer than any seen in training.

  5. Retrain with a scheduled-sampling rule that replaces the true character with the model's prediction with a probability that rises over training.

  6. Compare free-running accuracy for both models, especially on the longer strings where early mistakes have more steps to compound.

Mit mutat

The learner sees near-perfect accuracy for the teacher-forced model when given true prefixes, and a lower score when it runs on its own outputs, especially on long strings. The comparison shows exposure bias directly and whether scheduled sampling narrows the gap on this task.

Valós megvalósítás

Large language models are pretrained by next-token prediction with teacher forcing, the training setup that Bachmann and Nagarajan's 2024 paper on the pitfalls of next-token prediction analyzes in Transformer and Mamba models.

The PyTorch sequence-to-sequence translation tutorial uses teacher forcing in its decoder, noting that it speeds convergence but can make the trained network unstable when run on its own outputs.

Google researchers used scheduled sampling, a gradual move away from teacher forcing, in the system that ranked first in the 2015 MSCOCO image captioning challenge.

The original 2017 Transformer translation model trains its decoder on the target sentence shifted right by one position, with a mask hiding later words, so every target word is predicted from the true earlier words.

Professor Forcing, presented at NIPS 2016, trained recurrent models for handwriting, character-level text and raw-waveform speech so their behavior with and without teacher forcing looked alike.

A tanárkényszer jövője a sorozatmodellekben

Teacher forcing remains the core of language model pretraining, so its limits attract steady research attention. One line asks whether next-token prediction under teacher forcing can learn tasks that require planning ahead. Bachmann and Nagarajan (ICML 2024, revised in July 2025) give preliminary evidence that a teacherless objective, which uses dummy tokens so the model must predict several future tokens without seeing them, can avoid the failure they identify. Another line revisits the idea behind Ranzato and colleagues' sequence-level training: closing the train-inference gap by scoring and learning from the model's own complete outputs rather than only from true prefixes. Open questions include how much exposure bias actually harms large modern models compared with other error sources, and which training objectives best support long-horizon reasoning. Published results do not yet settle whether any alternative will replace teacher-forced next-token prediction as the default pretraining objective.

A gyakorlatba ültetés

Miért számít

  • Költség és költségvetés. Az építészeti döntések évekig növelik a teljesítményt és a működési költségeket.
  • Tisztább döntések. A technikai oktatás segít a csapatoknak a megfelelő verem kiválasztásában, nem csak a legújabb készletben.
  • Minőségellenőrzés. A jobb mérnöki döntések csökkentik a termelés megbízhatósági incidenseit.

Vigyázz

  • Egy benchmark optimalizálása elrejtheti a rendszer általános hiányosságait.
  • Az infrastrukturális és karbantartási költségeket gyakran alábecsülik.
  • A biztonsági és megfigyelhetőségi hiányosságok a rendszerek bonyolultabbá válásával nőhetnek.

Következő lépések

  1. Határozza meg a késleltetési, minőségi és költségcélokat a megvalósítás előtt.
  2. Benchmark reális terhelési és adatviszonyok mellett.
  3. Műszerfigyelés a hibák, az eltolódás és a felhasználói hatások szempontjából.
  4. A méretezés előtt készítse elő a visszagörgetési és az incidensre adott válaszútvonalakat.

Források és további olvasmányok

  1. Neural Computation (via ML Anthology)A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
  2. arXivScheduled Sampling for Sequence Prediction with Recurrent Neural Networks
  3. arXiv (ICLR 2016)Sequence Level Training with Recurrent Neural Networks
  4. arXiv (NIPS 2016)Professor Forcing: A New Algorithm for Training Recurrent Networks
  5. arXiv (NeurIPS 2017)Csak a figyelem kell
  6. PyTorchNLP From Scratch: Translation with a Sequence to Sequence Network and Attention
  7. arXiv (ICML 2024)The pitfalls of next-token prediction

Folytassa a felfedezést

Ingyenes hírlevél

Szerezze meg a napi AI összefoglalót

Minden hétköznap reggel három ellenőrzött AI történet, egyszerű angol nyelven. Örökké ingyen, hirdetések nélkül.

Minden hétköznap egy e-mail. Egy kattintással lemondhat. Soha nem értékesítjük vagy osztjuk meg az Ön címét.

Teszteld magad

Take the Teacher Forcing in Sequence Models quiz

Azonnali visszajelzés minden válaszra, és megosztható tanúsítvány egy ellenőrizhető azonosítóval, miután sikeresen teljesít egy kurzust.

Kezdő kvíz

Támogassa az ingyenes AI oktatást. AI Understanding egy 501(c)(3) nonprofit – nincs hirdetés, nincs fizetőfal, soha. Tegyen adományt

Gyakran ismételt kérdések

What is Teacher Forcing in Sequence Models?

Teacher forcing is the standard way to train models that generate sequences one token at a time: during training, the model is fed the correct previous token rather than its own guess. Named by Williams and Zipser in 1989, it makes training fast and stable and lets Transformers train on whole sequences in parallel, but it creates a gap between training and generation that researchers still study.

A tanári kényszerítés során mi kerül bemenetként minden dekódolási lépésnél?

A tanár kényszerítése a valódi előző cél tokent táplálja, nem pedig a modell előrejelzését, így a kondicionáló előtag helyesen marad.

During teacher-forced training, what target is paired with decoder input y_(t−1)?

With teacher forcing, the decoder receives y_(t−1) as context and the cross-entropy target is the true next token y_t.

A Transformer dekóderben milyen mechanizmus teszi lehetővé, hogy a tanárok által kényszerített képzés párhuzamosan működjön a különböző pozíciókban?

Az ok-okozati maszk megakadályozza, hogy az egyes pozíciók jövőbeli tokenekhez kapcsolódjanak, így az egész sorozat egyszerre feldolgozható csalás nélkül.

A következtetés idején miért nem használható a tanári kényszer?

A valós generálás során nem létezik célszekvencia, így a modellnek vissza kell táplálnia a saját előrejelzéseit, ellentétben a képzés során.

Melyik veszteséget használják jellemzően az egyes lépésekben a tanári kényszerképzés során?

Az autoregresszív modelleket token szintű keresztentrópiával képezik, összehasonlítva az előre jelzett eloszlást a következő arany tokennel.