引導式測驗 · 硬 等級
RMSNorm 和預層標準化測驗
檢查 RMSNorm 統計資料、Pre-LN 佈局、剩餘路徑和查詢鍵規範化,而不會混淆這些獨立的設計選擇。
問題 1 的 8
與標準 LayerNorm 相比,RMSNorm 省略了什麼?
預覽每個問題8 問題
當您參加測驗時,就會出現答案和解釋。
- 與標準 LayerNorm 相比,RMSNorm 省略了什麼?
- 平均減法(重新置中)
- 學習到的尺度參數
- 剩餘連接
- Softmax 運算
- RMSNorm 將每個激活向量除以什麼數量?
- 其方差
- 其均方根
- 其最大絕對值
- 它的L1範數
- Pre-LN 在深度 Transformer 初始化時支援什麼樣的梯度行為?
- It guarantees stable training for every model without tuning
- It removes the residual stream from the block
- It can make gradients better behaved at initialization and reduce some stability problems
- It makes all gradients exactly equal across layers
- 在 Pre-LN 區塊中,歸一化層故意保持不變的路徑是什麼?
- 前饋權重
- 令牌嵌入
- 輸出softmax
- 剩餘(身分)連接
- 哪些現代開放權重模型系列預設使用 RMSNorm?
- LLaMA、米斯特拉爾和奎文
- AlexNet 與 VGG
- ResNet 和 Inception
- BERT-base 和原始 Transformer
- 什麼是 RMSNorm 的最新應用「QK-norm」?
- 標準化最終輸出 logits
- 將 RMSNorm 應用於注意力查詢和鍵以控制 logit 增長
- 量化鍵值緩存
- 一種完全跳過標準化的方法
- A feature vector has a nonzero mean before RMSNorm. What does the normalization do to that mean?
- It always sets the mean exactly to zero
- It replaces each feature with its rank
- It does not explicitly subtract the mean, though rescaling can change its value
- It subtracts the maximum activation from every feature
- How are RMSNorm and Pre-LN related as design choices?
- They are two names for subtracting the mean
- RMSNorm always means normalization after the residual
- Pre-LN specifies the optimizer rather than the block
- RMSNorm specifies the normalization calculation; Pre-LN specifies where normalization sits in the residual block