AI Glossary Term
What does “Positional Encoding” mean?
Definition
Information added to token embeddings so transformers can distinguish sequence order.
How Positional Encoding works
Example
Without positions, a transformer would treat 'dog bites man' and 'man bites dog' as the same bag of words. Adding position information to each token tells it which came first.
Why it matters
Word order carries meaning, and attention by itself ignores it. Different encoding methods affect how well models handle long texts.
Common questions
Original method?
The 2017 transformer used fixed sine and cosine patterns added to embeddings.
Modern alternatives?
Learned positions and relative methods such as RoPE and ALiBi.
Why does it affect context length?
Models can struggle with positions longer than those seen in training.