Sliding Window Attention Quiz

Answers and explanations appear as you take the quiz.

1. With fixed window W, how does the pairwise attention term N times W scale as sequence length N increases?
   • Quadratically in N despite W being fixed
   • Exponentially in the vocabulary size by definition
   • It never changes as any tokens are added
   • Linearly in N
2. How can stacked local attention layers propagate information beyond one layer’s direct neighborhood?
   • By guaranteeing perfect recall of every earlier token
   • Through intermediate representations passed across successive layers
   • By randomly moving the original input’s coordinates
   • By eliminating the need for all learned weights
3. For a purely local causal attention layer, why can cache policy discard keys older than its active window?
   • Those keys are outside that layer’s allowed direct attention links
   • Every model layer must therefore have an identical whole-model memory bound
   • Older tokens are always irrelevant to every possible task
   • The model’s original weights have been deleted
4. What attention pattern does the Gemma 2 report describe across its layers?
   • Only one permanently disconnected input token
   • No positional or attention computation anywhere
   • Alternating local sliding-window and global attention
   • A mandatory single identical local neighborhood for all models
5. For a causal sliding window of size W, which keys can a query at position i attend to?
   • All future positions after i
   • Positions i minus W plus 1 through i
   • Only position 0
   • A random subset of all positions
6. In the Mistral report’s rolling cache, which slot stores the keys and values for timestep i with buffer size W?
   • A new permanent slot numbered i+W with no reuse
   • Only slot zero for every timestep
   • The slot at i modulo W
   • A slot chosen by the output token’s alphabetical order
7. Which context_len value is listed in the original Mistral 7B architecture table, separate from its larger theoretical stacked attention span?
   • 4096 tokens
   • 131072 tokens
   • 32 tokens
   • 8192 tokens
8. In Longformer’s stated symmetric global-attention design, how does a selected global token connect to other sequence tokens?
   • It attends to them and they can attend to it
   • It can receive no attention from any other token
   • It must ignore every token outside its local window
   • It requires all non-global tokens to be deleted