本頁閱讀時間 9 分鐘
概述
It subtracts a fixed, distance-proportional penalty from every attention score, which let models trained on short sequences run on longer ones and was adopted by BLOOM and MosaicML's MPT models.
重點摘要
- ALiBi encodes position by subtracting a fixed penalty proportional to token distance from attention scores instead of adding position embeddings to tokens.
- Each attention head uses a different fixed slope from a geometric sequence, such as 1/2 through 1/256 for an 8-head model.
- The 2021 paper reports a 1.3B model trained on 1,024 tokens matched a sinusoidal model trained on 2,048 tokens while training 11% faster.
- BigScience's BLOOM and MosaicML's MPT-7B both use ALiBi, and MPT-7B-StoryWriter generated up to 84,000 tokens in MosaicML's tests.
- ALiBi helps models run past their training length without collapsing, but its authors found it may not actually use the extra context.
深入探討
ALiBi, short for Attention with Linear Biases, is a method for giving a transformer language model a sense of word order. A transformer's attention layer compares every token with every other token, but the comparison itself ignores where tokens sit in the sequence, so models need some extra signal about position. ALiBi supplies that signal in the simplest possible way: when one token attends to another, the attention score is reduced by an amount that grows in a straight line with the distance between them. Nearby tokens are penalized a little, distant tokens a lot, and no position vectors are added to the token embeddings at all.
The mechanism works step by step as follows. First, the model computes the usual attention score for a query token at position i and a key token at position j, which is the dot product of the query and key vectors. Second, ALiBi adds a bias equal to a head-specific slope m multiplied by the negative distance, so the key one step back gets minus m, two steps back gets minus 2m, and so on, while the token attending to itself gets zero. Third, the softmax turns the adjusted scores into attention weights as normal. Because the slope differs between attention heads, some heads fall off steeply and focus on the last few tokens while others decay slowly and look much further back. The slopes are fixed before training rather than learned, and the bias is applied to query-key scores in every layer but never to the value vectors.
ALiBi was introduced by Ofir Press, Noah A. Smith and Mike Lewis in the paper Train Short, Test Long, first posted to arXiv on August 27, 2021 and revised in April 2022. The motivating question was extrapolation: can a model trained on sequences of one length make good predictions on longer sequences at inference time? The paper showed that the standard sinusoidal position embeddings from the original transformer break down almost immediately past the training length. On WikiText-103, a sinusoidal model trained on 512 tokens rose from 20.05 perplexity at 512 tokens to 132.41 at 2,512 tokens, while an ALiBi model trained on 512 tokens reached 18.40 at 3,072 tokens, better than a sinusoidal model trained at the full 3,072. Perplexity is a measure of how surprised a model is by the next token, so lower is better.
The practical selling point was cost. Training on short sequences is cheaper because attention cost grows with sequence length. The paper's larger experiment trained 1.3-billion-parameter models on 461 GB of CC100 and RoBERTa text using 128 V100 GPUs, and reports that an ALiBi model trained on 1,024-token inputs matched a sinusoidal model trained on 2,048 tokens while training 11% faster and using 11% less memory. The authors also report the method adds almost nothing at runtime: on one V100, training throughput and memory at 1,024 tokens were within about 1% of the sinusoidal baseline, and the per-head bias mask costs up to roughly 100 MB.
Adoption followed quickly. BigScience's BLOOM, the 176-billion-parameter open multilingual model whose training began on March 11, 2022 on 384 A100 GPUs, lists ALiBi as its positional encoding in its official model card. MosaicML's MPT-7B, released on May 5, 2023 and pretrained on one trillion tokens, also dropped position embeddings for ALiBi. Its StoryWriter variant was fine-tuned with a 65,000-token context, and MosaicML reported it handled about 68,000 tokens of The Great Gatsby and produced generations up to 84,000 tokens, which it attributed to ALiBi's extrapolation.
The obvious alternatives are absolute embeddings, relative biases and rotary embeddings. Absolute embeddings, sinusoidal or learned, attach a vector for each position number to the token, which ties the model to the positions it saw in training. The T5 relative bias learns a bias per bucket of distances; in the ALiBi paper it extrapolated better than sinusoidal and was the strongest baseline, but trained at least twice as slowly. Rotary Position Embedding (RoPE), proposed by Jianlin Su and colleagues in April 2021, rotates query and key vectors by an angle set by position so that scores depend on relative distance; it extrapolated better than sinusoidal in the ALiBi comparison but not as well as ALiBi. RoPE is the scheme used by models such as LLaMA, and methods such as Position Interpolation, posted by Chen and colleagues in June 2023, extend those RoPE models to 32,768 tokens with fewer than 1,000 fine-tuning steps.
ALiBi has real limits, several stated by its own authors. Their sliding-window analysis found that much of the gain at long evaluation lengths comes from reducing the early token curse, the poor predictions a model makes near the start of a chunk where it has little preceding context, rather than from actually using information beyond the training length. In other words, ALiBi lets a model run on longer inputs without collapsing, but it may not read the extra text in a useful way. On the 1.3B experiments, ALiBi's perplexity was best near twice the training length and only stayed usable to around 10,000 tokens. The slopes were picked from about ten manual trials on one dataset, learned slopes extrapolated poorly, and most experiments used a 247-million-parameter model. A 2023 NeurIPS study by Kazemnejad and colleagues found that on downstream reasoning-style tasks, ALiBi, rotary and absolute embeddings all generalized poorly to longer lengths, and a decoder with no positional encoding did better.
In practice ALiBi requires very little. It adds no trainable parameters, only a fixed matrix of distance penalties per head that can be precomputed, and the slope formula is a single line of code. The main engineering requirement is an attention kernel that accepts an additive bias, which matters because fused kernels such as FlashAttention need explicit support for it; MPT shipped with FlashAttention and ALiBi together. The method suits teams that want cheaper short-context pretraining with graceful behavior on longer inputs, while teams that need a model to retrieve facts from far back in a very long document should test long-context recall directly rather than rely on low perplexity alone.
04工作範例
Computing ALiBi biases for a tiny 4-head model
Pick the number of heads, n = 4, and compute the slope base 2^(-8/4) = 2^(-2) = 0.25.
List the head slopes as a geometric sequence with that first term and ratio: 0.25, 0.0625, 0.015625 and 0.00390625.
Take a query at position 5 attending to keys at positions 1 through 5, so the distances are 4, 3, 2, 1 and 0.
For head 1, multiply each distance by -0.25 to get biases of -1.0, -0.75, -0.5, -0.25 and 0.
For head 4, multiply by -0.00390625 to get biases of about -0.016, -0.012, -0.008, -0.004 and 0.
Add each head's bias vector to that head's raw query-key scores, apply the causal mask, then take the softmax.
它顯示了什麼
Head 1 now strongly favors the most recent tokens while head 4 is almost neutral about distance. Check the work by confirming every bias is zero on the diagonal, negative elsewhere, and grows in magnitude linearly with distance.
現實世界的實施
BigScience's BLOOM, a 176-billion-parameter open multilingual model trained in 2022, lists ALiBi as its positional encoding in its Hugging Face model card, with a 2,048-token training sequence length.
MosaicML's MPT-7B family, released May 5, 2023, replaced position embeddings with ALiBi; the StoryWriter variant was fine-tuned at 65k tokens and generated up to 84k tokens in the company's tests.
Cutting pretraining cost: the ALiBi paper reports a 1.3B-parameter model trained on 1,024-token inputs matching a sinusoidal model trained on 2,048 tokens, while training 11% faster with 11% less memory.
Research baselines for length generalization, such as the NeurIPS 2023 study by Kazemnejad and colleagues, which compares ALiBi against rotary, T5-style relative bias, absolute embeddings and no positional encoding at all.
The Future of ALiBi Position Bias: Attention with Linear Biases
The central open problem ALiBi exposed is the gap between running on long inputs and using them. Its own authors concluded that ALiBi may not exploit context beyond the training length, and the 2023 NeurIPS study of positional encodings by Kazemnejad and colleagues found ALiBi, rotary and absolute embeddings all generalize poorly on longer downstream tasks, with no positional encoding performing better in their setup. That result keeps alive the question of how much explicit position information decoder-only models need at all. A separate line of work extends rotary-embedding models instead, with techniques such as Position Interpolation (June 2023) stretching LLaMA models to 32,768 tokens after short fine-tuning. ALiBi remains in widely used open checkpoints, notably BLOOM and the MPT family, and its core idea of a cheap, fixed recency bias continues to appear as a baseline in length-generalization research. Whether future long-context models favor distance biases, rotation-based schemes, hybrids, or no explicit positions is an active research question rather than a settled one, and any claim about long-context quality is best checked with retrieval and reasoning tests at the target length.
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
資料來源與延伸閱讀
- arXiv (Press, Smith, Lewis)Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- BigScience on Hugging Facebigscience/bloom model card
- MosaicML / DatabricksIntroducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs
- arXiv (Su et al.)RoFormer: Enhanced Transformer with Rotary Position Embedding
- arXiv / NeurIPS 2023 (Kazemnejad et al.)The Impact of Positional Encoding on Length Generalization in Transformers
- arXiv (Chen et al.)Extending Context Window of Large Language Models via Positional Interpolation
不斷探索
常見問題
What is ALiBi Position Bias: Attention with Linear Biases?
ALiBi (Attention with Linear Biases) is a way to tell a transformer where each token sits without adding position embeddings. It subtracts a fixed, distance-proportional penalty from every attention score, which let models trained on short sequences run on longer ones and was adopted by BLOOM and MosaicML's MPT models.
ALiBi 代表什麼?
ALiBi 是 Attention with Linear Biases 的縮寫,因其添加到註意力分數中的線性懲罰而得名。
ALiBi如何懲罰遠程代幣?
ALiBi 從注意力分數中減去與查詢鍵距離成比例的值,因此距離越遠的標記權重越小。
ALiBi最著名的實際優勢是什麼?
由於偏差僅取決於距離,因此在短序列上訓練的模型可以在推理時處理更長的序列。
注意力頭之間的線性偏差有什麼不同?
ALiBi 為每個頭分配一個不同的固定斜率(例如 1/2、1/4、1/8),因此不同的頭部參與不同的範圍。
與傳統的位置嵌入相比,ALiBi 增加了多少學習參數?
ALiBi 沒有引入新的學習參數;人均斜率是預先決定的常數。