Block-Sparse and Native Sparse Attention Quiz

Answers and explanations appear as you take the quiz.

1. Why is standard self-attention expensive for long sequences?
   • Its cost grows quadratically with sequence length
   • It uses too few parameters
   • It cannot run on GPUs
   • It only attends to the first token
2. What is the core idea of block-sparse attention?
   • Removing the feed-forward layers
   • Computing attention only between selected blocks of tokens
   • Using one attention head only
   • Doubling the embedding dimension
3. What makes Native Sparse Attention (NSA) distinct from many earlier sparse methods?
   • It removes attention entirely
   • It only works during inference
   • It is trainable end-to-end and hardware-aligned
   • It requires no GPU
4. Which three branches does NSA combine for its keys and values?
   • Encoder, decoder, and embedding
   • Dropout, normalization, and pooling
   • Quantization, pruning, and distillation
   • Compression, selection, and a sliding window
5. Why does NSA use block-level operations rather than arbitrary token-level sparsity?
   • To align with GPU memory access and tensor-core throughput for real speedups
   • To make the model larger
   • To avoid using attention at all
   • To reduce the number of layers
6. During autoregressive decoding, sparse attention is especially helpful because that step is often:
   • Compute-bound on matrix multiplies
   • Memory-bound, so reducing attended tokens speeds it up
   • Independent of sequence length
   • Run only on CPUs
7. What information does NSA’s compression branch provide to the attention computation?
   • A full dense score for every token pair
   • Only the optimizer’s first moment
   • Coarse-grained summaries of blocks of tokens
   • A random subset of output labels
8. Why can learned sparsity patterns be more useful than a fixed arbitrary mask?
   • They remove the need to compute attention scores
   • They guarantee zero memory use
   • They force every token to attend globally
   • They can select blocks relevant to the input while remaining aligned with efficient hardware operations