GUIDA TECNICA

Parallelismo di sequenze e attenzione all'anello

Sequence parallelism splits one long input sequence across several GPUs or TPUs along the token dimension, so no single device has to hold the whole context.

Ascolta questa guidaNarrato dall'IA · ~8 min

In questa pagina9 minuti di lettura
  1. Panoramica
  2. Punti chiave
  3. Immersione profonda
  4. Tracing Ring Attention on four devices
  5. Implementazione nel mondo reale
  6. Il futuro del parallelismo delle sequenze e dell'attenzione dell'anello
  7. Metterlo in pratica
  8. Fonti e approfondimenti
  9. Continua a esplorare
  10. Domande frequenti

Panoramica

Ring Attention, published by UC Berkeley researchers in 2023, computes exact attention in that setting by passing key and value blocks around a ring of devices while each device keeps computing, which made million-token training contexts practical.

Punti chiave

  1. Sequence parallelism splits one sequence across devices along the token dimension so each device stores only part of the context.
  2. Ring Attention passes key and value blocks around a ring of devices and overlaps that transfer with blockwise attention computation.
  3. Ring Attention computes exact attention, not an approximation, by combining blockwise results with a running softmax.
  4. The Ring Attention paper reports over 1 million tokens of context for a 7B model on 32 A100 GPUs.
  5. DeepSpeed-Ulysses instead uses all-to-all communication so each GPU computes full-sequence attention for a subset of heads.
  6. Hiding communication requires blocks of at least FLOPS divided by bandwidth tokens, about 1,000 on NVLink-connected A100s.

Immersione profonda

A transformer processes a sequence of tokens, and its attention step lets every token look at every other token. That makes memory grow quickly with context length. The Ring Attention paper gives a concrete example: processing 100 million tokens with batch size 1 needs over 1,000 GB of memory even for a modest model with a hidden size of 1,024, while GPUs and TPUs typically have less than 100 GB of high-bandwidth memory. Sequence parallelism is the family of techniques that split a single sequence along its length, the token dimension, across several accelerators, so each one stores only a slice. It complements the older forms of parallelism: data parallelism splits the batch, tensor parallelism splits weight matrices, and pipeline parallelism splits layers.

Most transformer operations are easy to split by token, because they treat each token independently: layer normalization, the feed-forward network, dropout and the query, key and value projections. Attention is the exception. Each token's query must be compared with the keys of every other token in the sequence, and its output is a weighted sum of their values. If device 1 holds tokens 1 to 1,000 and device 2 holds tokens 1,001 to 2,000, device 2's queries still need device 1's keys and values. Every sequence-parallel method is really an answer to one question: how to get keys and values, or queries, to the right device without stalling the computation.

Ring Attention's answer is to arrange the devices in a ring. Each device keeps its own block of queries for the whole computation. Key and value blocks travel around the ring: at each step a device computes attention between its queries and the key-value block it currently holds, while simultaneously sending that block to the next device and receiving a new one from the previous device. After as many steps as there are devices, every query block has seen every key-value block. Because attention is computed blockwise with a running softmax, the same technique used by memory-efficient attention kernels, the partial results can be combined exactly, so the output is identical to ordinary attention, not an approximation. If each block's computation takes longer than its transfer, the communication is fully hidden.

The idea has a short history. Shenggui Li and colleagues at the National University of Singapore proposed sequence parallelism with Ring Self-Attention in May 2021; compared with tensor parallelism on 64 NVIDIA P100 GPUs, they reported 13.7 times larger maximum batch size and 3.0 times longer maximum sequence length. In May 2022, NVIDIA's Vijay Korthikanti and colleagues used the name sequence parallelism for a narrower technique inside Megatron-LM: splitting the LayerNorm and dropout regions along the sequence and swapping all-reduce operations for all-gather and reduce-scatter. Combined with selective recomputation, it cut activation memory by 5 times. In October 2023, Hao Liu, Matei Zaharia and Pieter Abbeel at UC Berkeley published Ring Attention with Blockwise Transformers, which fully overlapped communication with blockwise attention and feed-forward computation. In February 2024 the same group released Large World Model, a family of 7B models handling text and video beyond 1 million tokens.

The main alternative is DeepSpeed-Ulysses from Microsoft, published in September 2023. It also starts with the sequence split across GPUs, but just before attention it performs an all-to-all exchange so that each GPU receives the full sequence for a subset of attention heads, computes those heads, and then reverses the exchange. Its authors show that communication volume stays constant when sequence length and device count grow together, and report up to 2.5 times higher throughput with 4 times longer sequences than the prior state of the art. Because work is divided by heads, the number of devices it can use for one sequence is bounded by the number of attention heads, while Ring Attention divides by blocks of tokens. Megatron Core's context parallelism follows the Ring Attention pattern and adds cuDNN flash attention kernels and causal-mask load balancing.

The headline results are large. On 32 A100 GPUs, the Ring Attention paper reports over 1 million tokens of context for a 7B model, a 32 times improvement over the previous best memory-efficient transformer, and on TPUv4-512 a 256 times increase, allowing training sequences of over 30 million tokens. In general, Ring Attention allows sequences up to device-count times longer than memory-efficient transformers on a single device. Large World Model used it to train on TPUv4-1024, which its authors describe as roughly equivalent to 450 A100 GPUs, scaling RoPE positional embeddings as context grew.

There are real limitations. Hiding communication needs enough work per block: the Ring Attention paper derives that the block size must be at least the device's FLOPS divided by its interconnect bandwidth, which is about 1,000 tokens for A100s on NVLink or TPUs, but about 24,500 tokens for A100s linked by InfiniBand at 12.5 GB per second. Causal language models create load imbalance, because with contiguous blocks the device holding the first tokens has almost nothing to attend to while the last device does the most work; Striped Attention fixes this by giving each device tokens spread throughout the sequence, reporting up to 1.45 times speedups at 256K tokens on A100s and TPUs and 1.65 times at 786K tokens on 16 TPUv4 chips. Sequence parallelism also does not reduce the total compute of attention, which still grows with the square of sequence length, so a million-token step remains expensive.

In practice, using these methods requires a framework that supports them, such as Megatron Core with Transformer Engine, DeepSpeed, or a JAX implementation like the one published with Ring Attention, plus fast device-to-device links. Total GPU count becomes the product of all parallel dimensions; Megatron Core's documentation states it as tensor times context times pipeline times data parallel size. Training a long-context model also involves more than the parallelism scheme: the Large World Model authors describe data curation, progressive context extension and positional-embedding scaling as necessary parts of the process.

04Esempio lavorato

Tracing Ring Attention on four devices

  1. Take a 16,000-token sequence and four devices, giving each device a contiguous block of 4,000 tokens with its own queries, keys and values.

  2. At step 1, each device computes attention between its queries and its own key-value block, while sending that block to the next device in the ring.

  3. At step 2, each device receives the previous device's key-value block, updates its running maximum, exponential sum and weighted value sum, and forwards the block again.

  4. Repeat for steps 3 and 4, so every query block has met all four key-value blocks.

  5. Divide each device's weighted value sum by its exponential sum to get the final attention output for its 4,000 tokens.

  6. For a causal model, note that the device holding tokens 1 to 4,000 can skip most blocks while the last device must process all four, which is the imbalance Striped Attention addresses.

Cosa mostra

Exact attention outputs for all 16,000 tokens, with each device storing only about six blocks of 4,000 tokens. Check it by running the same input through a single-device attention implementation on a small example and comparing outputs, which should match to numerical precision.

Implementazione nel mondo reale

UC Berkeley's Large World Model project used Blockwise RingAttention to extend a LLaMA-2 7B model progressively from 4K to 1M tokens of context, training on a TPUv4-1024 pod.

NVIDIA Megatron Core offers context parallelism, which it describes as similar to Ring Attention but with cuDNN flash attention kernels and load balancing for causal masks, combined with tensor, pipeline and data parallelism.

Microsoft's DeepSpeed-Ulysses splits sequences across GPUs and uses all-to-all communication to give each GPU full sequences for a subset of attention heads, reporting training on sequences over a million tokens.

NVIDIA's Megatron-LM sequence parallelism splits LayerNorm and dropout activations along the sequence; with selective recomputation it trained a 530B-parameter GPT-3 style model at 54.2% model FLOPs utilization on 2,240 A100 GPUs.

Striped Attention (MIT, 2023) reassigns tokens to devices in a strided pattern to balance causal-attention work, reporting up to 1.45x faster training than Ring Attention at 256K tokens.

Il futuro del parallelismo delle sequenze e dell'attenzione dell'anello

Work since 2023 has concentrated on three problems. Load balance for causal models is one: Striped Attention (November 2023) and Megatron Core's context parallelism both reorganize which tokens each device holds so that no device sits idle behind the causal mask. Communication is another: DeepSpeed-Ulysses (September 2023) showed that all-to-all exchanges can keep per-link volume constant, and frameworks increasingly combine head-based and ring-based splitting. Kernel efficiency is the third, with context-parallel implementations calling fused flash attention kernels for each block. The broader open question is how much long context helps compared with its cost. Ring Attention removes the per-device memory limit but not the quadratic compute of exact attention, so training on million-token sequences still consumes large accelerator budgets, and the Large World Model paper reports that data curation and positional-embedding scaling were as important as parallelism. Research into efficient attention variants, retrieval and memory mechanisms continues alongside these systems methods, and published benchmarks of throughput and long-context retrieval accuracy remain the reliable way to compare them.

Metterlo in pratica

Perché è importante

  • Costo e budget. Le decisioni relative all'architettura determinano prestazioni e costi operativi per anni.
  • Decisioni più chiare. La formazione tecnica aiuta i team a scegliere lo stack giusto, non solo quello più nuovo.
  • Controllo di qualità. Migliori scelte ingegneristiche riducono gli incidenti legati all’affidabilità nella produzione.

Attenzione

  • L'ottimizzazione di un benchmark può nascondere debolezze di sistema più ampie.
  • I costi delle infrastrutture e della manutenzione sono spesso sottostimati.
  • Le lacune in termini di sicurezza e osservabilità possono aumentare man mano che i sistemi diventano più complessi.

Prossimi passi

  1. Definire obiettivi di latenza, qualità e costi prima dell'implementazione.
  2. Benchmark in condizioni di carico e dati realistiche.
  3. Monitoraggio dello strumento per errori, deriva e impatto sull'utente.
  4. Preparare percorsi di rollback e risposta agli incidenti prima della scalabilità.

Fonti e approfondimenti

  1. arXiv (UC Berkeley)Ring Attention with Blockwise Transformers for Near-Infinite Context
  2. arXiv (National University of Singapore)Sequence Parallelism: Long Sequence Training from System Perspective
  3. arXiv (NVIDIA)Reducing Activation Recomputation in Large Transformer Models
  4. arXiv (Microsoft)DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models
  5. arXiv (ICLR 2025)World Model on Million-Length Video and Language with Blockwise RingAttention
  6. arXivStriped Attention: Faster Ring Attention for Causal Transformers
  7. NVIDIA Megatron Core documentationcontext_parallel package

Continua a esplorare

Newsletter gratuita

Ricevi il briefing quotidiano sull'IA

Tre storie verificate sull'IA ogni mattina nei giorni feriali, scritte in un inglese semplice. Gratuito per sempre, senza pubblicità.

Un'email ogni giorno feriale. Disiscriviti con un clic. Non vendiamo né condividiamo mai il tuo indirizzo.

Mettiti alla prova

Take the Sequence Parallelism and Ring Attention quiz

Feedback immediato su ogni risposta e un certificato condivisibile con un ID verificabile una volta superato un corso.

Inizia il quiz

Sostieni l'istruzione gratuita sull'IA. AI Understanding è un'organizzazione non profit 501(c)(3) — nessun annuncio, nessun paywall, mai. Fai una donazione

Domande frequenti

Cos'è il parallelismo di sequenza e l'attenzione dell'anello?

Sequence parallelism splits one long input sequence across several GPUs or TPUs along the token dimension, so no single device has to hold the whole context. Ring Attention, published by UC Berkeley researchers in 2023, computes exact attention in that setting by passing key and value blocks around a ring of devices while each device keeps computing, which made million-token training contexts practical.

Lungo quale dimensione il parallelismo delle sequenze suddivide i dati?

Il parallelismo delle sequenze suddivide una singola sequenza tra le GPU lungo l'asse token/tempo, in modo che ciascun dispositivo possieda un blocco contiguo di token.

Cosa passa l'attenzione dell'anello tra le GPU disposte in un anello?

Ogni GPU mantiene fisse le proprie query locali e passa i blocchi chiave/valore hop-by-hop attorno all'anello in modo che ogni query alla fine veda ogni chiave.

How can Ring Attention combine partial attention blocks without approximating the full attention pattern?

Online softmax accumulates the same attention expression across blocks. Floating-point reduction order can still produce small numerical differences.

Perché Ring Attention non richiede che una singola GPU memorizzi l'intero K/V?

Poiché i blocchi K/V arrivano in modo incrementale e ciascuna GPU accumula attenzione parziale, nessun dispositivo ha mai bisogno dell'intero set di chiavi/valori in memoria contemporaneamente.

When can Ring Attention reduce the exposed time spent transferring K/V blocks?

Overlap can hide transfer time when the computation and bandwidth permit it. It does not guarantee communication-free execution on every layout.