FlashAttention Quiz

Answers and explanations appear as you take the quiz.

1. What is the main bottleneck FlashAttention targets in standard attention?
   • Reading and writing the large attention matrix to slow GPU memory
   • The number of floating-point multiplications
   • The cost of tokenizing the input text
   • Network latency between GPUs
2. How does FlashAttention's output compare to standard attention?
   • It is an approximation that trades accuracy for speed
   • It is mathematically identical (exact)
   • It only works for short sequences
   • It changes the model's weights
3. Which GPU memory does FlashAttention exploit by processing data in tiles?
   • The large but slow high-bandwidth memory (HBM)
   • The CPU's RAM
   • The small but very fast on-chip SRAM
   • Disk storage
4. What technique lets FlashAttention compute softmax without seeing all scores at once?
   • Skipping the softmax entirely
   • Rounding all scores to zero
   • Precomputing softmax on the CPU
   • Online (incremental) softmax with running rescaling
5. Because the full attention matrix is never stored in HBM, memory usage scales roughly:
   • Linearly with sequence length
   • Quadratically with sequence length
   • Exponentially with sequence length
   • Independently of sequence length
6. Which numeric format does FlashAttention-3 use in its Hopper-focused experiments to accelerate attention?
   • Only 64-bit floating point
   • FP8, with techniques to manage its numerical error
   • Integer text tokens
   • No reduced-precision format
7. What work-partitioning change does FlashAttention-2 introduce to improve GPU parallelism?
   • It stores the complete attention matrix in HBM
   • It replaces attention with convolution
   • It reduces non-matrix-multiplication work and improves parallelization across thread blocks and warps
   • It uses only CPU execution
8. In FlashAttention-3, what lets data movement overlap with computation?
   • Replacing attention with a lookup table
   • Computing all scores in host memory
   • Disabling the softmax step
   • Asynchronous execution on Hopper hardware