引導式測驗 · 中等 等級

滑動視窗注意力測驗

查看本地註意力掩碼和固定視窗縮放,以及堆疊感受野、滾動快取、Gemma 2 層和 Longformer 全域標記。

相關引導路徑滑動視窗注意
問題 1 的 8

在固定視窗 W 下,隨著序列長度 N 增加,兩兩注意力項目 N × W 的規模如何變化?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. 在固定視窗 W 下,隨著序列長度 N 增加,兩兩注意力項目 N × W 的規模如何變化?
    • Quadratically in N despite W being fixed
    • Exponentially in the vocabulary size by definition
    • It never changes as any tokens are added
    • Linearly in N
  2. 堆疊的局部注意力層如何將資訊傳遞至超出單層直接鄰域的範圍?
    • By guaranteeing perfect recall of every earlier token
    • Through intermediate representations passed across successive layers
    • By randomly moving the original input’s coordinates
    • By eliminating the need for all learned weights
  3. 對於純粹的局部因果注意力層,為何快取政策可以丟棄超出其活躍視窗的舊鍵?
    • Those keys are outside that layer’s allowed direct attention links
    • Every model layer must therefore have an identical whole-model memory bound
    • Older tokens are always irrelevant to every possible task
    • The model’s original weights have been deleted
  4. Gemma 2 報告描述的注意力模式在各層之間是什麼?
    • Only one permanently disconnected input token
    • No positional or attention computation anywhere
    • Alternating local sliding-window and global attention
    • A mandatory single identical local neighborhood for all models
  5. 對於大小為 W 的因果滑動窗口,i 位置的查詢可以關注哪些按鍵?
    • i 之後的所有未來職位
    • 位置 i 減 W 加 1 到 i
    • 僅位置0
    • 所有位置的隨機子集
  6. In the Mistral report’s rolling cache, which slot stores the keys and values for timestep i with buffer size W?
    • A new permanent slot numbered i+W with no reuse
    • Only slot zero for every timestep
    • The slot at i modulo W
    • A slot chosen by the output token’s alphabetical order
  7. Which context_len value is listed in the original Mistral 7B architecture table, separate from its larger theoretical stacked attention span?
    • 4096 tokens
    • 131072 tokens
    • 32 tokens
    • 8192 tokens
  8. In Longformer’s stated symmetric global-attention design, how does a selected global token connect to other sequence tokens?
    • It attends to them and they can attend to it
    • It can receive no attention from any other token
    • It must ignore every token outside its local window
    • It requires all non-global tokens to be deleted