AI Glossary Term

What does “RoPE (Rotary Positional Embeddings)” mean?

Definition

A positional encoding method that rotates query and key vectors to encode relative token positions.

Browse the full AI GlossaryExplore all free AI guides

How RoPE (Rotary Positional Embeddings) works

Diagram for RoPE (Rotary Positional Embeddings), steps in order: Query and key vectors; then Rotate by an angle that depends on position; then Dot product reflects relative distance; then Attention knows how far apart tokens are.Query and key vectorsRotate by an angle that depends onpositionDot product reflects relative distanceAttention knows how far apart tokensare

Example

Two tokens three places apart get rotated by amounts whose difference depends only on that gap. Attention can then favor nearby tokens in a way that works wherever in the text the pair appears.

Why it matters

RoPE is used in many popular open models and supports techniques that extend context length. It encodes relative position directly in attention.

Common questions

Who proposed RoPE?

Su and colleagues in the 2021 RoFormer paper.

Can RoPE models handle longer text than they trained on?

Not automatically, but scaling tricks can extend usable context, often with some fine-tuning.

RoPE vs absolute encoding?

Absolute encoding marks each token's place; RoPE encodes how far apart tokens are.

Related terms

Positional EncodingInformation added to token embeddings so transformers can distinguish sequence order.ALiBiA positional bias method that penalizes attention scores based on token distance, helping models extrapolate to longer contexts.RAG-FusionA retrieval method that generates multiple query variants, retrieves results for each, and fuses the rankings.ANN (Approximate Nearest Neighbor)Algorithms that find vectors close to a query without exhaustive comparison, trading exactness for speed.RAG (Retrieval-Augmented Generation)A method that retrieves external knowledge and feeds it into generation at inference time.Reinforcement Learning from Human Feedback (RLHF)A training method that uses human preference signals to shape model behavior.

Learn more in our free guides

Audio Embeddings and Representation LearningSentence-BERT EmbeddingsX-Vector Speaker EmbeddingsHyDE Hypothetical Document Embeddings

Test yourself

Audio Embeddings and Representation Learning QuizSentence-BERT Embeddings QuizX-Vector Speaker Embeddings Quiz

See also

Multi-Head AttentionSliding Window AttentionFlash AttentionBPE (Byte Pair Encoding)QLoRASentencePieceDPO (Direct Preference Optimization)Tree of Thoughts