引導式測驗 · 中等 等級

旋轉位置嵌入測驗

評估 RoPE 如何旋轉查詢和關鍵向量來編碼位置,並將其相對位置屬性與長上下文縮放方法區分開來。

相關引導路徑旋轉位置嵌入
問題 1 的 8

RoPE 如何將位置資訊注入變壓器?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. RoPE 如何將位置資訊注入變壓器?
    • 透過將查詢向量和鍵向量中的維度對旋轉位置相關的角度
    • 透過向輸入嵌入添加固定向量
    • 透過隨機重新排序令牌
    • 透過將位置編號附加為文字
  2. 在應用RoPE後,兩個token之間的注意力分數主要取決於:
    • 僅他們的絕對位置
    • 他們的相對距離(m減n)
    • 詞彙量大小
    • 批量大小
  3. What training-efficiency comparison did the YaRN paper report for context extension?
    • Exactly the same data and steps as all prior methods
    • No fine-tuning or training data of any kind
    • About 10x fewer tokens and 2.5x fewer training steps than prior methods
    • A fixed 100x increase in output tokens
  4. 為什麼不同維度對在 RoPE 中以不同頻率旋轉?
    • 透過跳過某些維度來節省內存
    • 隨機化模型的輸出
    • 因為所有對必須使用相同的角度
    • 捕捉精細的局部秩序和遠程位置
  5. What is the goal of techniques such as Position Interpolation and YaRN for RoPE-based models?
    • Extend the usable context length beyond the original training window using additional scaling/fine-tuning methods
    • Guarantee accurate recall at any sequence length with no evaluation
    • Remove position information from attention
    • Replace all token embeddings with absolute learned vectors
  6. Does standard RoPE require learned absolute-position vectors to inject position into query and key representations?
    • Yes; it adds one learned vector for every possible sequence position
    • No; it applies position-dependent rotations to paired query/key dimensions
    • It removes query and key vectors entirely
    • It only adds position to value vectors
  7. What does standard RoPE alone guarantee about a model’s context length?
    • Unlimited accurate context at every position
    • A fixed context of exactly 128 tokens
    • It does not by itself guarantee reliable use beyond the trained context length
    • Removal of the attention mechanism
  8. What extended context length did the Position Interpolation paper report for its LLaMA evaluations?
    • Exactly 1,000 tokens with no position changes
    • Unlimited context with no fine-tuning
    • Up to 2 million tokens for all models
    • Up to 32,768 tokens with minimal fine-tuning