Sparse Attention Patterns Quiz

Answers and explanations appear as you take the quiz.

1. Why is full self-attention expensive on long sequences?
   • Its cost grows with the square of the sequence length
   • It cannot use GPUs
   • It only works on short words
   • It requires no memory at all
2. What is sliding-window (local) attention?
   • Each token attends to a random database
   • Each token attends only to its nearby neighbors
   • Tokens never attend to anything
   • Each token attends to all tokens twice
3. What is the purpose of 'global tokens' in sparse attention?
   • To slow the model down
   • To store images
   • To act as hubs that attend to everything and that everything attends to
   • To replace the vocabulary
4. Which model proved that combining random, window, and global attention keeps full-attention expressiveness?
   • ResNet
   • Word2Vec
   • AlexNet
   • BigBird
5. Roughly how does the number of connections scale in well-designed sparse attention?
   • Roughly linearly with sequence length
   • Quadratically with sequence length
   • It does not depend on length
   • Exponentially with sequence length
6. What do strided or dilated sparse-attention patterns allow?
   • Computes them and then sets them to zero
   • Skip ahead to reach distant context with fewer attention links
   • Sends them to another server
   • Stores them permanently
7. How does Longformer combine local and global attention?
   • Every token attends to every other token
   • Only random edges are allowed
   • Most tokens use a sliding local window, while selected global tokens connect across the sequence
   • Each token attends only to the final token
8. What is the goal of learned, content-based sparse attention?
   • A full dense connection to every token
   • A rule for removing all future tokens
   • A learned vocabulary for each attention head
   • Avoid spending attention compute and memory on content unrelated to the current query