Induction Heads in Transformers
Induction heads are attention heads that implement a simple but powerful copy rule: 'I saw [A][B] earlier, and now I see [A] again, so predict [B].' They are a key mechanism behind transformers' striking ability to do in-context learning from just a few examples in the prompt.
Deep Dive
Discovered through mechanistic interpretability of small transformers, induction heads emerge during training at a characteristic moment that lines up with a sudden drop in loss and the onset of in-context learning. They typically work as a two-head circuit. A 'previous-token head' in an earlier layer copies information about each token's predecessor forward. Then the induction head uses that to perform prefix matching: it finds an earlier occurrence of the current token, looks at what followed it, and attends back to copy that next token into the prediction. This pattern-completion ability lets models repeat sequences, complete analogies, and pick up novel formats or word definitions defined entirely within the prompt, without any weight updates.
Technical Insight
The circuit is a composition of two attention heads across layers. The previous-token head writes 'the token before me was X' into each position's residual stream. The induction head's query-key matching (Q-K) then matches the current token against those shifted keys to locate prior [A] positions, and its output-value path (O-V) copies the token that followed. This is a concrete example of cross-layer 'K-composition' studied in transformer circuits research.
Strategic Impact
Speed and scale
Language workflows can move faster without sacrificing consistency.
Access and reach
It expands access across languages and communication styles.
Clearer decisions
Teams can spend more time on judgment while automation handles repetition.
The Future of Induction Heads in Transformers
Induction heads are a flagship success of mechanistic interpretability, and the field is extending the idea to richer 'in-context learning circuits' that handle abstraction, not just literal copying. Expect more work linking the abrupt formation of these heads to phase changes and emergent abilities in larger models. Understanding when and how such circuits form could help predict capabilities, design better curricula, and build safety tools that detect when models are learning unintended behaviors purely from context.
Real-World Implementation
Completing a repeated random token sequence like 'A B C ... A B' by predicting 'C' from earlier context.
Few-shot prompting where the model copies the input-output format demonstrated in earlier examples.
Learning a made-up word's meaning given in the prompt and reusing it correctly later in the same passage.
Faithfully echoing a long quoted string or list by matching prior occurrences of its tokens.
Risks & Guardrails
Hallucinated facts can quietly enter reports, support flows, or research outputs.
Prompt sensitivity can create inconsistent results across similar requests.
Sensitive text data may be exposed if access controls are weak.
Implementation Roadmap
Define output format, tone, and quality standards before rollout.
Ground responses with trusted sources whenever accuracy matters.
Keep a human review checkpoint for high-stakes outputs.
Track failure patterns and retrain prompts or workflows regularly.
Keep Exploring
Free newsletter
Get the daily AI briefing
Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.
One email each weekday. Unsubscribe in one click. We never sell or share your address.
Test yourself
Take the Induction Heads in Transformers quiz
Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.
Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation
Next guide
Jamba Hybrid Transformer-Mamba Models
Frequently asked questions
What is Induction Heads in Transformers?
Induction heads are attention heads that implement a simple but powerful copy rule: 'I saw [A][B] earlier, and now I see [A] again, so predict [B].' They are a key mechanism behind transformers' striking ability to do in-context learning from just a few examples in the prompt.
What rule does an induction head essentially implement?
Induction heads do prefix matching: they find where the current token appeared before and copy whatever followed it.
Induction heads are most closely associated with which capability?
Their emergence coincides with the onset of in-context learning, the ability to adapt from examples in the prompt without weight updates.
What role does the 'previous-token head' play in the induction circuit?
The previous-token head writes 'my predecessor was X' into the residual stream, enabling the induction head to match and copy.
How many attention heads are typically involved in the canonical induction circuit?
The classic circuit is a two-head composition: a previous-token head plus the induction head that performs matching and copying.
What is notable about when induction heads form during training?
Induction heads appear at a phase-change-like moment aligned with a sudden loss drop and the emergence of in-context learning.