What does Multi-Head Attention mean? Definition | AI Understanding

Definition

A transformer mechanism that runs several attention operations in parallel to capture different types of relationships.

Related terms

A neural architecture that uses attention to model relationships across sequences in parallel.

Multi-Query Retrieval

A retrieval technique that rewrites the user query into several variants to improve recall.

Attention Mechanism

A model component that dynamically focuses on relevant parts of an input when producing an output.

Flash Attention

An optimized attention algorithm that reduces memory use and speeds up transformer training and inference.

Multimodal Model

A model that can process or generate multiple data types such as text, image, and audio.

Knowledge Graph

A graph structure of entities and relationships used for reasoning or retrieval.

Learn more in our free guides

Multilingual LLMs Multimodal Search Hard Parameter Sharing in Multi-Task Networks Attention Rollout and Head Pruning

See also

Positional Encoding QLoRA RoPE (Rotary Positional Embeddings)DPO (Direct Preference Optimization)ALiBi Tree of Thoughts Sliding Window Attention ReAct

Browse the full AI Glossary Explore all free AI guides