引導式測驗 · 中等 等級
滑動視窗注意力測驗
查看本地註意力掩碼和固定視窗縮放,以及堆疊感受野、滾動快取、Gemma 2 層和 Longformer 全域標記。
問題 1 的 8
在固定視窗 W 下,隨著序列長度 N 增加,兩兩注意力項目 N × W 的規模如何變化?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 在固定視窗 W 下,隨著序列長度 N 增加,兩兩注意力項目 N × W 的規模如何變化?
- Quadratically in N despite W being fixed
- Exponentially in the vocabulary size by definition
- It never changes as any tokens are added
- Linearly in N
- 堆疊的局部注意力層如何將資訊傳遞至超出單層直接鄰域的範圍?
- By guaranteeing perfect recall of every earlier token
- Through intermediate representations passed across successive layers
- By randomly moving the original input’s coordinates
- By eliminating the need for all learned weights
- 對於純粹的局部因果注意力層,為何快取政策可以丟棄超出其活躍視窗的舊鍵?
- Those keys are outside that layer’s allowed direct attention links
- Every model layer must therefore have an identical whole-model memory bound
- Older tokens are always irrelevant to every possible task
- The model’s original weights have been deleted
- Gemma 2 報告描述的注意力模式在各層之間是什麼?
- Only one permanently disconnected input token
- No positional or attention computation anywhere
- Alternating local sliding-window and global attention
- A mandatory single identical local neighborhood for all models
- 對於大小為 W 的因果滑動窗口,i 位置的查詢可以關注哪些按鍵?
- i 之後的所有未來職位
- 位置 i 減 W 加 1 到 i
- 僅位置0
- 所有位置的隨機子集
- In the Mistral report’s rolling cache, which slot stores the keys and values for timestep i with buffer size W?
- A new permanent slot numbered i+W with no reuse
- Only slot zero for every timestep
- The slot at i modulo W
- A slot chosen by the output token’s alphabetical order
- Which context_len value is listed in the original Mistral 7B architecture table, separate from its larger theoretical stacked attention span?
- 4096 tokens
- 131072 tokens
- 32 tokens
- 8192 tokens
- In Longformer’s stated symmetric global-attention design, how does a selected global token connect to other sequence tokens?
- It attends to them and they can attend to it
- It can receive no attention from any other token
- It must ignore every token outside its local window
- It requires all non-global tokens to be deleted