Flash Attention Quiz

Answers and explanations appear as you take the quiz.

1. What is the main bottleneck that Flash Attention targets?
   • Reading and writing the large attention matrix to slow GPU memory
   • The number of floating-point multiplications in attention
   • The cost of loading model weights from disk
   • Network latency between multiple GPUs
2. How does Flash Attention avoid storing the full N-by-N attention matrix?
   • It approximates attention with random projections
   • It processes queries, keys, and values in small tiles and combines partial results
   • It skips most token pairs entirely
   • It compresses the matrix with quantization
3. What technique lets Flash Attention compute softmax correctly without seeing the whole row at once?
   • A precomputed lookup table of exponentials
   • Dropping the softmax and using a linear approximation
   • A running (online) softmax that tracks a running max and sum
   • Sorting scores before normalizing
4. Compared to standard attention, the output of Flash Attention is:
   • A rough approximation that trades accuracy for speed
   • More accurate because of higher precision
   • Only valid for short sequences
   • Mathematically identical, just computed more efficiently
5. Why does Flash Attention help most with long sequences?
   • Because attention memory grows quadratically with sequence length
   • Because longer sequences need fewer layers
   • Because softmax becomes unnecessary for long inputs
   • Because GPUs run slower on short sequences
6. What does the 'IO-aware' design of Flash Attention prioritize minimizing?
   • The number of model parameters
   • Memory reads and writes between SRAM and HBM
   • The depth of the neural network
   • The vocabulary size of the tokenizer
7. What remains computationally quadratic for dense exact self-attention even when FlashAttention reduces memory traffic?
   • The number of model layers must double
   • Every token is compared with only one other token
   • The number of query-key pair interactions grows with the square of sequence length
   • The number of model parameters becomes zero
8. Why can FlashAttention’s results differ slightly from a reference implementation despite computing the same mathematical attention?
   • It intentionally drops all long-distance tokens
   • It changes the model weights during inference
   • It replaces softmax with a random sample
   • Floating-point operations can occur in a different order during tiled accumulation