技術指南

序列並行性與環注意力

序列平行化將一長輸入序列沿著 token 維度拆分到多個 GPU 或 TPU 上,這樣單一裝置不必承載整個上下文。

聆聽此指南AI 語音敘述 · ~8 min

本頁閱讀時間 9 分鐘
  1. 概述
  2. 重點摘要
  3. 深入探討
  4. 在四台設備上追蹤 Ring Attention
  5. 現實世界的實施
  6. 序列並行性與環注意力的未來
  7. 付諸實踐
  8. 資料來源與延伸閱讀
  9. 不斷探索
  10. 常見問題

概述

加州大學柏克萊分校研究人員於 2023 年發布的 Ring Attention,透過在設備環周圍傳遞鍵和值塊,同時每個設備保持計算,來計算該設置中的精確注意力,這使得百萬令牌訓練環境變得實用。

重點摘要

  1. 序列並行性沿著令牌維度將一個序列拆分到裝置之間,因此每個裝置僅儲存上下文的一部分。
  2. 環形注意力在裝置環周圍傳遞鍵和值塊,並與區塊注意力計算進行重疊傳輸。
  3. Ring Attention 透過將分塊結果與運行的 softmax 結合來計算精確的注意力,而不是近似值。
  4. Ring Attention 論文報告了 32 個 A100 GPU 上 7B 模型的超過 100 萬個上下文標記。
  5. DeepSpeed-Ulysses 則相反地使用全對全通信,因此每個 GPU 計算頭部子集的全序列注意力。
  6. 隱藏通訊需要至少 FLOPS 除以頻寬令牌的區塊,在 NVLink 連接的 A100 上大約為 1,000。

深入探討

A transformer processes a sequence of tokens, and its attention step lets every token look at every other token. That makes memory grow quickly with context length. The Ring Attention paper gives a concrete example: processing 100 million tokens with batch size 1 needs over 1,000 GB of memory even for a modest model with a hidden size of 1,024, while GPUs and TPUs typically have less than 100 GB of high-bandwidth memory. Sequence parallelism is the family of techniques that split a single sequence along its length, the token dimension, across several accelerators, so each one stores only a slice. It complements the older forms of parallelism: data parallelism splits the batch, tensor parallelism splits weight matrices, and pipeline parallelism splits layers.

Most transformer operations are easy to split by token, because they treat each token independently: layer normalization, the feed-forward network, dropout and the query, key and value projections. Attention is the exception. Each token's query must be compared with the keys of every other token in the sequence, and its output is a weighted sum of their values. If device 1 holds tokens 1 to 1,000 and device 2 holds tokens 1,001 to 2,000, device 2's queries still need device 1's keys and values. Every sequence-parallel method is really an answer to one question: how to get keys and values, or queries, to the right device without stalling the computation.

Ring Attention's answer is to arrange the devices in a ring. Each device keeps its own block of queries for the whole computation. Key and value blocks travel around the ring: at each step a device computes attention between its queries and the key-value block it currently holds, while simultaneously sending that block to the next device and receiving a new one from the previous device. After as many steps as there are devices, every query block has seen every key-value block. Because attention is computed blockwise with a running softmax, the same technique used by memory-efficient attention kernels, the partial results can be combined exactly, so the output is identical to ordinary attention, not an approximation. If each block's computation takes longer than its transfer, the communication is fully hidden.

The idea has a short history. Shenggui Li and colleagues at the National University of Singapore proposed sequence parallelism with Ring Self-Attention in May 2021; compared with tensor parallelism on 64 NVIDIA P100 GPUs, they reported 13.7 times larger maximum batch size and 3.0 times longer maximum sequence length. In May 2022, NVIDIA's Vijay Korthikanti and colleagues used the name sequence parallelism for a narrower technique inside Megatron-LM: splitting the LayerNorm and dropout regions along the sequence and swapping all-reduce operations for all-gather and reduce-scatter. Combined with selective recomputation, it cut activation memory by 5 times. In October 2023, Hao Liu, Matei Zaharia and Pieter Abbeel at UC Berkeley published Ring Attention with Blockwise Transformers, which fully overlapped communication with blockwise attention and feed-forward computation. In February 2024 the same group released Large World Model, a family of 7B models handling text and video beyond 1 million tokens.

The main alternative is DeepSpeed-Ulysses from Microsoft, published in September 2023. It also starts with the sequence split across GPUs, but just before attention it performs an all-to-all exchange so that each GPU receives the full sequence for a subset of attention heads, computes those heads, and then reverses the exchange. Its authors show that communication volume stays constant when sequence length and device count grow together, and report up to 2.5 times higher throughput with 4 times longer sequences than the prior state of the art. Because work is divided by heads, the number of devices it can use for one sequence is bounded by the number of attention heads, while Ring Attention divides by blocks of tokens. Megatron Core's context parallelism follows the Ring Attention pattern and adds cuDNN flash attention kernels and causal-mask load balancing.

The headline results are large. On 32 A100 GPUs, the Ring Attention paper reports over 1 million tokens of context for a 7B model, a 32 times improvement over the previous best memory-efficient transformer, and on TPUv4-512 a 256 times increase, allowing training sequences of over 30 million tokens. In general, Ring Attention allows sequences up to device-count times longer than memory-efficient transformers on a single device. Large World Model used it to train on TPUv4-1024, which its authors describe as roughly equivalent to 450 A100 GPUs, scaling RoPE positional embeddings as context grew.

There are real limitations. Hiding communication needs enough work per block: the Ring Attention paper derives that the block size must be at least the device's FLOPS divided by its interconnect bandwidth, which is about 1,000 tokens for A100s on NVLink or TPUs, but about 24,500 tokens for A100s linked by InfiniBand at 12.5 GB per second. Causal language models create load imbalance, because with contiguous blocks the device holding the first tokens has almost nothing to attend to while the last device does the most work; Striped Attention fixes this by giving each device tokens spread throughout the sequence, reporting up to 1.45 times speedups at 256K tokens on A100s and TPUs and 1.65 times at 786K tokens on 16 TPUv4 chips. Sequence parallelism also does not reduce the total compute of attention, which still grows with the square of sequence length, so a million-token step remains expensive.

In practice, using these methods requires a framework that supports them, such as Megatron Core with Transformer Engine, DeepSpeed, or a JAX implementation like the one published with Ring Attention, plus fast device-to-device links. Total GPU count becomes the product of all parallel dimensions; Megatron Core's documentation states it as tensor times context times pipeline times data parallel size. Training a long-context model also involves more than the parallelism scheme: the Large World Model authors describe data curation, progressive context extension and positional-embedding scaling as necessary parts of the process.

04工作範例

在四台設備上追蹤 Ring Attention

  1. 採用 16,000 個令牌序列和四個設備,為每個設備提供一個由 4,000 個令牌組成的連續區塊,並具有自己的查詢、鍵和值。

  2. 在步驟 1 中,每個裝置計算其查詢與其自己的鍵值區塊之間的注意力,同時將該區塊傳送到環中的下一個裝置。

  3. 步驟2,每個設備接收前一個設備的鍵值塊,更新其運行最大值、指數和和加權值和,並再次轉發該塊。

  4. 重複步驟 3 和 4,以便每個查詢區塊都滿足所有四個鍵值區塊。

  5. 將每個設備的權重值總和除以其指數總和,以獲得其 4,000 個令牌的最終注意力輸出。

  6. 對於因果模型,請注意,持有代幣 1 到 4,000 的設備可以跳過大多數區塊,而最後一個設備必須處理所有四個區塊,這就是 Striped Attention 解決的不平衡問題。

它顯示了什麼

所有 16,000 個令牌的精確注意力輸出,每個裝置僅儲存約 6 個區塊,每區塊 4,000 個令牌。透過在一個小範例上透過單一裝置注意力實現運行相同的輸入並比較輸出來檢查它,輸出應與數值精度相符。

現實世界的實施

加州大學柏克萊分校的大世界模型計畫使用 Blockwise RingAttention 將 LLaMA-2 7B 模型逐步從 4K 上下文標記擴展到 1M 上下文標記,並在 TPUv4-1024 pod 上進行訓練。

NVIDIA Megatron Core 提供上下文並行性,它被描述為類似於環注意,但具有 cuDNN flash 注意內核和因果掩碼的負載平衡,並與張量、管道和資料並行性相結合。

Microsoft 的 DeepSpeed-Ulysses 在 GPU 之間分割序列,並使用全對所有通訊為每個 GPU 提供注意力頭子集的完整序列,報告超過一百萬個令牌的序列訓練。

NVIDIA 的 Megatron-LM 序列並行性沿著序列分割 LayerNorm 和 dropout 活化;透過選擇性重新計算,它在 2,240 個 A100 GPU 上以 54.2% 的模型 FLOP 利用率訓練了 530B 參數的 GPT-3 樣式模型。

Striped Attention(MIT,2023)以跨步模式將令牌重新分配給設備,以平衡因果注意力工作,報告稱,在 256K 令牌下,訓練速度比 Ring Attention 快 1.45 倍。

序列並行性與環注意力的未來

Work since 2023 has concentrated on three problems. Load balance for causal models is one: Striped Attention (November 2023) and Megatron Core's context parallelism both reorganize which tokens each device holds so that no device sits idle behind the causal mask. Communication is another: DeepSpeed-Ulysses (September 2023) showed that all-to-all exchanges can keep per-link volume constant, and frameworks increasingly combine head-based and ring-based splitting. Kernel efficiency is the third, with context-parallel implementations calling fused flash attention kernels for each block. The broader open question is how much long context helps compared with its cost. Ring Attention removes the per-device memory limit but not the quadratic compute of exact attention, so training on million-token sequences still consumes large accelerator budgets, and the Large World Model paper reports that data curation and positional-embedding scaling were as important as parallelism. Research into efficient attention variants, retrieval and memory mechanisms continues alongside these systems methods, and published benchmarks of throughput and long-context retrieval accuracy remain the reliable way to compare them.

付諸實踐

為什麼這很重要

  • 成本與預算. 多年來,架構決策決定著效能和營運成本。
  • 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
  • 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。

留意

  • 優化一項基準測試可以隱藏更廣泛的系統弱點。
  • 基礎設施和維護成本常常被低估。
  • 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。

後續步驟

  1. 在實施之前定義延遲、品質和成本目標。
  2. 在實際負載和資料條件下進行基準測試。
  3. 儀器監控錯誤、漂移和使用者影響。
  4. 在擴展之前準備回滾和事件回應路徑。

資料來源與延伸閱讀

  1. arXiv (UC Berkeley)使用 Blockwise Transformer 實現近乎無限上下文的環注意力機制
  2. arXiv (National University of Singapore)序列平行:系統視角的長序列訓練
  3. arXiv (NVIDIA)減少大型變壓器模型中的啟動重新計算
  4. arXiv (Microsoft)DeepSpeed Ulysses:用於支援極長序列變壓器模型訓練的系統最佳化
  5. arXiv (ICLR 2025)使用 Blockwise RingAttention 的百萬長度影片和語言的世界模型
  6. arXivStriped Attention:Causal Transformer 的更快的 Ring Attention
  7. NVIDIA Megatron Core documentationcontext_parallel 套件

不斷探索

免費電子報

獲取每日 AI 簡報

每個工作日上午提供三則經過驗證的 AI 新聞,以淺顯易懂的英文撰寫。永久免費,無廣告。

每個工作日一封電子郵件。一鍵取消訂閱。我們絕不出售或分享您的地址。

測試一下自己

參加序列並行性與環注意力測驗

每個答案皆有即時回饋,通過課程後可獲得具備可驗證 ID 的可分享證書。

開始測驗

支持免費的 AI 教育。AI Understanding 是一個 501(c)(3) 非營利組織——絕無廣告,絕無付費牆。 捐款

常見問題

什麼是序列並行和環注意力?

序列並行性將一個長輸入序列沿著標記維度拆分到多個 GPU 或 TPU 上,因此沒有任何單一裝置必須保存整個上下文。加州大學柏克萊分校研究人員於 2023 年發布的 Ring Attention,透過在設備環周圍傳遞鍵和值塊,同時每個設備保持計算,來計算該設置中的精確注意力,這使得百萬令牌訓練環境變得實用。

序列並行沿著哪個維度分割資料?

序列並行性沿令牌/時間軸跨 GPU 分片單一序列,因此每個裝置都有連續的令牌塊。

環形注意力機制在排列成環形的 GPU 之間傳遞什麼?

每個 GPU 都保持其本地查詢固定,並在環上逐跳傳遞鍵/值區塊,因此每個查詢最終都會看到每個鍵。

環形注意力如何在不近似完整注意力模式的情況下組合部分注意力塊?

在線softmax跨區塊累積相同的注意力表達。浮點降階仍然會產生小的數值差異。

為什麼 Ring Attention 不需要任何單一 GPU 來儲存完整的 K/V?

由於 K/V 區塊增量到達並且每個 GPU 累積部分注意力,因此沒有裝置需要立即將整個鍵/值集儲存在記憶體中。

Ring Attention什麼時候可以減少傳輸K/V塊所花費的暴露時間?

當計算和頻寬允許時,重疊可以隱藏傳輸時間。它不保證在每個佈局上無通訊執行。