Transformers

Attention computes how much information one position should take from other positions. In a common formulation, learned projections produce queries, keys, and values. Query-key comparisons determine weights used to combine values. Multiple attention heads allow several such combinations within a layer.

A transformer layer also includes other operations, such as a feed-forward network, normalization, and residual connections. Position information is needed because the order of words or other sequence elements matters. Specific implementations differ in how they represent position and arrange these operations.

The original 2017 transformer used an encoder-decoder design for translation. Later models use encoder-only, decoder-only, or encoder-decoder arrangements for different objectives. A causal language model prevents a position from attending to future tokens during next-token prediction. That constraint differs from bidirectional processing of a complete input.

Attention over long sequences can be computationally expensive. Practical systems use varied optimizations, but an advertised context limit does not prove that the model uses every part of a long document reliably. Test retrieval, reasoning, and instruction following at the actual lengths your application needs.