AI Glossary Term
What does “Flash Attention” mean?
Definition
An optimized attention algorithm that reduces memory use and speeds up transformer training and inference.
How Flash Attention works
Example
Training a model on long documents is faster and fits in memory with Flash Attention because it processes attention in blocks and avoids storing the full attention matrix.
Why it matters
It makes longer contexts and faster training practical with no change to the math. It is a hardware-aware optimization.
Common questions
Does it change the results?
No. It computes the same attention output exactly, just more efficiently.
Who created it?
Tri Dao and colleagues, first published in 2022.
Why is memory movement the issue?
On modern GPUs, moving data often costs more time than the arithmetic itself.