引導式測驗 · 中等 等級
旋轉位置嵌入測驗
評估 RoPE 如何旋轉查詢和關鍵向量來編碼位置,並將其相對位置屬性與長上下文縮放方法區分開來。
問題 1 的 8
RoPE 如何將位置資訊注入變壓器?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- RoPE 如何將位置資訊注入變壓器?
- 透過將查詢向量和鍵向量中的維度對旋轉位置相關的角度
- 透過向輸入嵌入添加固定向量
- 透過隨機重新排序令牌
- 透過將位置編號附加為文字
- 在應用RoPE後,兩個token之間的注意力分數主要取決於:
- 僅他們的絕對位置
- 他們的相對距離(m減n)
- 詞彙量大小
- 批量大小
- What training-efficiency comparison did the YaRN paper report for context extension?
- Exactly the same data and steps as all prior methods
- No fine-tuning or training data of any kind
- About 10x fewer tokens and 2.5x fewer training steps than prior methods
- A fixed 100x increase in output tokens
- 為什麼不同維度對在 RoPE 中以不同頻率旋轉?
- 透過跳過某些維度來節省內存
- 隨機化模型的輸出
- 因為所有對必須使用相同的角度
- 捕捉精細的局部秩序和遠程位置
- What is the goal of techniques such as Position Interpolation and YaRN for RoPE-based models?
- Extend the usable context length beyond the original training window using additional scaling/fine-tuning methods
- Guarantee accurate recall at any sequence length with no evaluation
- Remove position information from attention
- Replace all token embeddings with absolute learned vectors
- Does standard RoPE require learned absolute-position vectors to inject position into query and key representations?
- Yes; it adds one learned vector for every possible sequence position
- No; it applies position-dependent rotations to paired query/key dimensions
- It removes query and key vectors entirely
- It only adds position to value vectors
- What does standard RoPE alone guarantee about a model’s context length?
- Unlimited accurate context at every position
- A fixed context of exactly 128 tokens
- It does not by itself guarantee reliable use beyond the trained context length
- Removal of the attention mechanism
- What extended context length did the Position Interpolation paper report for its LLaMA evaluations?
- Exactly 1,000 tokens with no position changes
- Unlimited context with no fine-tuning
- Up to 2 million tokens for all models
- Up to 32,768 tokens with minimal fine-tuning