Kvíz spojený s průvodcem · Střední úroveň

Kvíz pozornosti s posuvným oknem

Prohlédněte si místní masky pozornosti a škálování s pevným oknem s naskládanými receptivními poli, rolovacími mezipaměti, vrstvami Gemma 2 a globálními tokeny Longformer.

Související vodicí cestyPosuvné okno Pozor
Otázka 1 z 8

With fixed window W, how does the pairwise attention term N times W scale as sequence length N increases?

Náhled každé otázky8 otázky

Odpovědi a vysvětlení se zobrazují, jak budete v kvízu.

  1. With fixed window W, how does the pairwise attention term N times W scale as sequence length N increases?
    • Quadratically in N despite W being fixed
    • Exponentially in the vocabulary size by definition
    • It never changes as any tokens are added
    • Linearly in N
  2. How can stacked local attention layers propagate information beyond one layer’s direct neighborhood?
    • By guaranteeing perfect recall of every earlier token
    • Through intermediate representations passed across successive layers
    • By randomly moving the original input’s coordinates
    • By eliminating the need for all learned weights
  3. For a purely local causal attention layer, why can cache policy discard keys older than its active window?
    • Those keys are outside that layer’s allowed direct attention links
    • Every model layer must therefore have an identical whole-model memory bound
    • Older tokens are always irrelevant to every possible task
    • The model’s original weights have been deleted
  4. What attention pattern does the Gemma 2 report describe across its layers?
    • Only one permanently disconnected input token
    • No positional or attention computation anywhere
    • Alternating local sliding-window and global attention
    • A mandatory single identical local neighborhood for all models
  5. Pro kauzální posuvné okno o velikosti W, které klávesy lze dotazovat na pozici, které se mám věnovat?
    • Všechny budoucí pozice po i
    • Pozice i mínus W plus 1 až i
    • Pouze pozice 0
    • Náhodná podmnožina všech pozic
  6. In the Mistral report’s rolling cache, which slot stores the keys and values for timestep i with buffer size W?
    • A new permanent slot numbered i+W with no reuse
    • Only slot zero for every timestep
    • The slot at i modulo W
    • A slot chosen by the output token’s alphabetical order
  7. Which context_len value is listed in the original Mistral 7B architecture table, separate from its larger theoretical stacked attention span?
    • 4096 tokens
    • 131072 tokens
    • 32 tokens
    • 8192 tokens
  8. In Longformer’s stated symmetric global-attention design, how does a selected global token connect to other sequence tokens?
    • It attends to them and they can attend to it
    • It can receive no attention from any other token
    • It must ignore every token outside its local window
    • It requires all non-global tokens to be deleted