AI Glossary Term

What does “Multi-Head Attention” mean?

Definition

A transformer mechanism that runs several attention operations in parallel to capture different types of relationships.

Browse the full AI Glossary Explore all free AI guides

How Multi-Head Attention works

Example

In one layer, one head may track which noun a pronoun refers to, while another focuses on neighboring words. Their results are combined into a richer representation.

Why it matters

Multiple heads let the model look at different relationships at once. Heads tend to specialize, though not always in neat human terms.

Common questions

How many heads?

It varies, from a handful to a hundred or more in large models.

How are head outputs combined?

They are concatenated and passed through a learned linear layer.

Does each head really specialize?

Some do, but many are redundant, and interpreting them is an active research area.

TransformerA neural architecture that uses attention to model relationships across sequences in parallel.Multi-Query RetrievalA retrieval technique that rewrites the user query into several variants to improve recall.Attention MechanismA model component that dynamically focuses on relevant parts of an input when producing an output.Flash AttentionAn optimized attention algorithm that reduces memory use and speeds up transformer training and inference.Multimodal ModelA model that can process or generate multiple data types such as text, image, and audio.Knowledge GraphA graph structure of entities and relationships used for reasoning or retrieval.

Learn more in our free guides

Multilingual LLMs Multimodal Search Hard Parameter Sharing in Multi-Task Networks Attention Rollout and Head Pruning

Test yourself

Multilingual LLMs Quiz Multimodal Search Quiz Hard Parameter Sharing in Multi-Task Networks Quiz