InayofuataMwongozo unaofuata
Ufafanuzi wa Nafasi kwa Kiendelezi cha Muktadha
Lugha AI
Lugha AI MWONGOZO
ALiBi Position Bias: Attention with Linear Biases
ALiBi (Attention with Linear Biases) is a way to tell a transformer where each token sits without adding position embeddings.
Sikiliza mwongozo huuInaonyeshwa na AI · ~8 min
Katika ukurasa huudk 9 kusoma
Muhtasari
It subtracts a fixed, distance-proportional penalty from every attention score, which let models trained on short sequences run on longer ones and was adopted by BLOOM and MosaicML's MPT models.
Mambo muhimu ya kuchukua
- ALiBi encodes position by subtracting a fixed penalty proportional to token distance from attention scores instead of adding position embeddings to tokens.
- Each attention head uses a different fixed slope from a geometric sequence, such as 1/2 through 1/256 for an 8-head model.
- The 2021 paper reports a 1.3B model trained on 1,024 tokens matched a sinusoidal model trained on 2,048 tokens while training 11% faster.
- BigScience's BLOOM and MosaicML's MPT-7B both use ALiBi, and MPT-7B-StoryWriter generated up to 84,000 tokens in MosaicML's tests.
- ALiBi helps models run past their training length without collapsing, but its authors found it may not actually use the extra context.
Dive ya kina
ALiBi, short for Attention with Linear Biases, is a method for giving a transformer language model a sense of word order. A transformer's attention layer compares every token with every other token, but the comparison itself ignores where tokens sit in the sequence, so models need some extra signal about position. ALiBi supplies that signal in the simplest possible way: when one token attends to another, the attention score is reduced by an amount that grows in a straight line with the distance between them. Nearby tokens are penalized a little, distant tokens a lot, and no position vectors are added to the token embeddings at all.
The mechanism works step by step as follows. First, the model computes the usual attention score for a query token at position i and a key token at position j, which is the dot product of the query and key vectors. Second, ALiBi adds a bias equal to a head-specific slope m multiplied by the negative distance, so the key one step back gets minus m, two steps back gets minus 2m, and so on, while the token attending to itself gets zero. Third, the softmax turns the adjusted scores into attention weights as normal. Because the slope differs between attention heads, some heads fall off steeply and focus on the last few tokens while others decay slowly and look much further back. The slopes are fixed before training rather than learned, and the bias is applied to query-key scores in every layer but never to the value vectors.
ALiBi was introduced by Ofir Press, Noah A. Smith and Mike Lewis in the paper Train Short, Test Long, first posted to arXiv on August 27, 2021 and revised in April 2022. The motivating question was extrapolation: can a model trained on sequences of one length make good predictions on longer sequences at inference time? The paper showed that the standard sinusoidal position embeddings from the original transformer break down almost immediately past the training length. On WikiText-103, a sinusoidal model trained on 512 tokens rose from 20.05 perplexity at 512 tokens to 132.41 at 2,512 tokens, while an ALiBi model trained on 512 tokens reached 18.40 at 3,072 tokens, better than a sinusoidal model trained at the full 3,072. Perplexity is a measure of how surprised a model is by the next token, so lower is better.
The practical selling point was cost. Training on short sequences is cheaper because attention cost grows with sequence length. The paper's larger experiment trained 1.3-billion-parameter models on 461 GB of CC100 and RoBERTa text using 128 V100 GPUs, and reports that an ALiBi model trained on 1,024-token inputs matched a sinusoidal model trained on 2,048 tokens while training 11% faster and using 11% less memory. The authors also report the method adds almost nothing at runtime: on one V100, training throughput and memory at 1,024 tokens were within about 1% of the sinusoidal baseline, and the per-head bias mask costs up to roughly 100 MB.
Adoption followed quickly. BigScience's BLOOM, the 176-billion-parameter open multilingual model whose training began on March 11, 2022 on 384 A100 GPUs, lists ALiBi as its positional encoding in its official model card. MosaicML's MPT-7B, released on May 5, 2023 and pretrained on one trillion tokens, also dropped position embeddings for ALiBi. Its StoryWriter variant was fine-tuned with a 65,000-token context, and MosaicML reported it handled about 68,000 tokens of The Great Gatsby and produced generations up to 84,000 tokens, which it attributed to ALiBi's extrapolation.
The obvious alternatives are absolute embeddings, relative biases and rotary embeddings. Absolute embeddings, sinusoidal or learned, attach a vector for each position number to the token, which ties the model to the positions it saw in training. The T5 relative bias learns a bias per bucket of distances; in the ALiBi paper it extrapolated better than sinusoidal and was the strongest baseline, but trained at least twice as slowly. Rotary Position Embedding (RoPE), proposed by Jianlin Su and colleagues in April 2021, rotates query and key vectors by an angle set by position so that scores depend on relative distance; it extrapolated better than sinusoidal in the ALiBi comparison but not as well as ALiBi. RoPE is the scheme used by models such as LLaMA, and methods such as Position Interpolation, posted by Chen and colleagues in June 2023, extend those RoPE models to 32,768 tokens with fewer than 1,000 fine-tuning steps.
ALiBi has real limits, several stated by its own authors. Their sliding-window analysis found that much of the gain at long evaluation lengths comes from reducing the early token curse, the poor predictions a model makes near the start of a chunk where it has little preceding context, rather than from actually using information beyond the training length. In other words, ALiBi lets a model run on longer inputs without collapsing, but it may not read the extra text in a useful way. On the 1.3B experiments, ALiBi's perplexity was best near twice the training length and only stayed usable to around 10,000 tokens. The slopes were picked from about ten manual trials on one dataset, learned slopes extrapolated poorly, and most experiments used a 247-million-parameter model. A 2023 NeurIPS study by Kazemnejad and colleagues found that on downstream reasoning-style tasks, ALiBi, rotary and absolute embeddings all generalized poorly to longer lengths, and a decoder with no positional encoding did better.
In practice ALiBi requires very little. It adds no trainable parameters, only a fixed matrix of distance penalties per head that can be precomputed, and the slope formula is a single line of code. The main engineering requirement is an attention kernel that accepts an additive bias, which matters because fused kernels such as FlashAttention need explicit support for it; MPT shipped with FlashAttention and ALiBi together. The method suits teams that want cheaper short-context pretraining with graceful behavior on longer inputs, while teams that need a model to retrieve facts from far back in a very long document should test long-context recall directly rather than rely on low perplexity alone.
04Mfano uliofanya kazi
Computing ALiBi biases for a tiny 4-head model
Pick the number of heads, n = 4, and compute the slope base 2^(-8/4) = 2^(-2) = 0.25.
List the head slopes as a geometric sequence with that first term and ratio: 0.25, 0.0625, 0.015625 and 0.00390625.
Take a query at position 5 attending to keys at positions 1 through 5, so the distances are 4, 3, 2, 1 and 0.
For head 1, multiply each distance by -0.25 to get biases of -1.0, -0.75, -0.5, -0.25 and 0.
For head 4, multiply by -0.00390625 to get biases of about -0.016, -0.012, -0.008, -0.004 and 0.
Add each head's bias vector to that head's raw query-key scores, apply the causal mask, then take the softmax.
Nini inaonyesha
Head 1 now strongly favors the most recent tokens while head 4 is almost neutral about distance. Check the work by confirming every bias is zero on the diagonal, negative elsewhere, and grows in magnitude linearly with distance.
Utekelezaji wa Ulimwengu Halisi
BigScience's BLOOM, a 176-billion-parameter open multilingual model trained in 2022, lists ALiBi as its positional encoding in its Hugging Face model card, with a 2,048-token training sequence length.
MosaicML's MPT-7B family, released May 5, 2023, replaced position embeddings with ALiBi; the StoryWriter variant was fine-tuned at 65k tokens and generated up to 84k tokens in the company's tests.
Cutting pretraining cost: the ALiBi paper reports a 1.3B-parameter model trained on 1,024-token inputs matching a sinusoidal model trained on 2,048 tokens, while training 11% faster with 11% less memory.
Research baselines for length generalization, such as the NeurIPS 2023 study by Kazemnejad and colleagues, which compares ALiBi against rotary, T5-style relative bias, absolute embeddings and no positional encoding at all.
The Future of ALiBi Position Bias: Attention with Linear Biases
The central open problem ALiBi exposed is the gap between running on long inputs and using them. Its own authors concluded that ALiBi may not exploit context beyond the training length, and the 2023 NeurIPS study of positional encodings by Kazemnejad and colleagues found ALiBi, rotary and absolute embeddings all generalize poorly on longer downstream tasks, with no positional encoding performing better in their setup. That result keeps alive the question of how much explicit position information decoder-only models need at all. A separate line of work extends rotary-embedding models instead, with techniques such as Position Interpolation (June 2023) stretching LLaMA models to 32,768 tokens after short fine-tuning. ALiBi remains in widely used open checkpoints, notably BLOOM and the MPT family, and its core idea of a cheap, fixed recency bias continues to appear as a baseline in length-generalization research. Whether future long-context models favor distance biases, rotation-based schemes, hybrids, or no explicit positions is an active research question rather than a settled one, and any claim about long-context quality is best checked with retrieval and reasoning tests at the target length.
Kuiweka katika vitendo
Kwa nini ni muhimu
- Kasi na kiwango. Mitiririko ya kazi ya lugha inaweza kusonga kwa kasi zaidi bila kuacha uthabiti.
- Kufikia na kufikia. Inapanua ufikiaji katika lugha na mitindo ya mawasiliano.
- Maamuzi ya wazi zaidi. Timu zinaweza kutumia muda mwingi kufanya uamuzi huku otomatiki ikishughulikia marudio.
Jihadharini na
- Mambo ya ukweli yanaweza kuingiza ripoti kwa utulivu, mitiririko ya usaidizi, au matokeo ya utafiti.
- Usikivu wa haraka unaweza kuunda matokeo yasiyolingana katika maombi sawa.
- Data nyeti ya maandishi inaweza kufichuliwa ikiwa vidhibiti vya ufikiaji ni dhaifu.
Hatua zinazofuata
- Bainisha umbizo la towe, toni na viwango vya ubora kabla ya kusambaza.
- Majibu ya msingi na vyanzo vinavyoaminika wakati wowote usahihi ni muhimu.
- Weka ukaguzi wa ukaguzi wa kibinadamu kwa matokeo ya juu.
- Fuatilia mifumo ya kushindwa na fundisha tena vidokezo au mtiririko wa kazi mara kwa mara.
Vyanzo na kusoma zaidi
- arXiv (Press, Smith, Lewis)Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
- BigScience on Hugging Facebigscience/bloom model card
- MosaicML / DatabricksIntroducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs
- arXiv (Su et al.)RoFormer: Enhanced Transformer with Rotary Position Embedding
- arXiv / NeurIPS 2023 (Kazemnejad et al.)The Impact of Positional Encoding on Length Generalization in Transformers
- arXiv (Chen et al.)Extending Context Window of Large Language Models via Positional Interpolation
Endelea Kuchunguza
Jarida halali
Pata taarifa ya kila siku ya AI
Hadithi tatu za AI zilizothibitishwa kila asubuhi ya wiki, zimetengenezwa kwa Kiingereza rahisi. Bila malipo daima, hakuna matangazo.
Barua pepe moja kila siku ya wiki. Jiondoe kwa bonyeza moja. Hatutauza au kushiriki anwani yako.
Jijaribu mwenyewe
Take the ALiBi Position Bias: Attention with Linear Biases quiz
Maoni ya haraka juu ya kila jibu, na cheti kinachoweza kushirikishwa na kitambulisho kinachoweza kuthibitishwa baada ya kupita kozi.
Changia elimu ya AI bila malipo. AI Understanding ni 501(c)(3) isiyo na faida — hakuna matangazo, hakuna kizuizi cha malipo, daima. Toa mchango
Maswali yanayoulizwa mara kwa mara
What is ALiBi Position Bias: Attention with Linear Biases?
ALiBi (Attention with Linear Biases) is a way to tell a transformer where each token sits without adding position embeddings. It subtracts a fixed, distance-proportional penalty from every attention score, which let models trained on short sequences run on longer ones and was adopted by BLOOM and MosaicML's MPT models.
ALiBi inasimamia nini?
ALiBi ni kifupi cha Attention with Linear Biases, iliyopewa jina la adhabu ya mstari ambayo huongeza kwa alama za umakini.
ALiBi inaadhibu vipi tokeni za mbali?
ALiBi huondoa thamani sawia na umbali wa ufunguo wa hoja kutoka kwa alama ya umakini, ili tokeni za mbali zipate uzito mdogo.
Je, ni faida gani ya vitendo inayoadhimishwa zaidi ya ALiBi?
Kwa sababu upendeleo hutegemea umbali tu, mifano iliyofunzwa kwenye mlolongo mfupi inaweza kushughulikia ndefu zaidi kwa wakati wa uelekezaji.
Ni nini tofauti kuhusu upendeleo wa mstari kwenye vichwa vya umakini?
ALiBi hupeana kila kichwa mteremko tofauti na usiobadilika (k.m., 1/2, 1/4, 1/8), kwa hivyo vichwa tofauti huhudhuria katika safu tofauti.
Ikilinganishwa na upachikaji wa nafasi za kitamaduni, ALiBi inaongeza vigezo vingapi vya kujifunza?
ALiBi inatanguliza hakuna vigezo vipya vilivyojifunza; miteremko ya kila-kichwa imeamuliwa mapema.
Endelea kujifunza
Miongozo inayohusiana
Miongozo zaidi imechaguliwa kwa mada hii