AI Glossary Term
What does “Attention Mechanism” mean?
A model component that dynamically focuses on relevant parts of an input when producing an output.
How Attention Mechanism works
Example
When a language model reads 'The trophy didn't fit in the suitcase because it was too big', attention lets it weigh 'trophy' more heavily than 'suitcase' when deciding what 'it' refers to. The model learns such weightings from data rather than following hand-written rules.
Why it matters
Attention is the core idea behind transformers, the architecture behind most modern language models. It lets models use relevant context from anywhere in the input.
Common questions
Does attention mean the model is paying attention like a person?
No. It is a mathematical weighting of how much each part of the input contributes to each output, learned during training.
Why did attention replace older methods?
It handles long-range relationships better than earlier recurrent designs and can be computed in parallel, which suits modern hardware.
What is a downside of attention?
Standard attention cost grows quickly with input length, which is why researchers developed faster variants and limits on context size.