本頁閱讀時間 9 分鐘
概述
Introduced by Microsoft in 2019 and shipped in DeepSpeed, it and its PyTorch counterpart FSDP are how most teams now fit multi-billion-parameter models onto ordinary GPU clusters.
重點摘要
- Mixed-precision Adam needs about 16 bytes per parameter, so a 7.5-billion-parameter model needs roughly 120 GB of model state before activations.
- ZeRO stage 1 shards optimizer state, stage 2 also shards gradients, and stage 3 also shards the model weights themselves.
- Stages 1 and 2 cost the same communication as ordinary data parallelism, while stage 3 costs about 1.5 times as much.
- The 2019 ZeRO paper from Microsoft reported training models over 100 billion parameters on 400 GPUs at about 15 petaflops.
- PyTorch FSDP provides ZeRO-style full sharding natively, and ZeRO-Offload and ZeRO-Infinity extend it to CPU and NVMe memory.
深入探討
ZeRO, the Zero Redundancy Optimizer, is a way to train neural networks that are too big for one GPU's memory while keeping the simple programming model of data parallelism. In ordinary data parallelism, every GPU holds a full copy of the model, processes a different slice of the training batch, and the GPUs average their gradients after each step. That is easy to use but wasteful: with 64 GPUs, there are 64 identical copies of everything. ZeRO removes that duplication by giving each GPU responsibility for only a fraction of the training state, and fetching the rest from its peers just when it is needed. A sharded optimizer is the general name for this pattern; PyTorch's Fully Sharded Data Parallel (FSDP) is the best-known implementation outside Microsoft's DeepSpeed library.
The reason duplication hurts is that the weights are not the biggest item. The ZeRO paper does the accounting for mixed-precision training with the Adam optimizer, where each parameter needs 2 bytes for a 16-bit copy of the weight, 2 bytes for its 16-bit gradient, and 12 bytes of 32-bit optimizer state: a master copy of the weight plus Adam's two running averages, momentum and variance. That is 16 bytes per parameter, so a 7.5-billion-parameter model needs about 120 GB just for model state, before counting activations, the intermediate values saved for the backward pass. The optimizer state alone is three quarters of the total, which is why sharding it first gives the biggest win.
ZeRO works in three cumulative stages. Stage 1 partitions the optimizer states: each of N GPUs keeps Adam's state for only 1/N of the parameters and updates only that slice, after which the updated weights are shared with everyone. Stage 2 also partitions the gradients: instead of every GPU receiving the full averaged gradient, a reduce-scatter operation sums gradients across GPUs and delivers each GPU only the slice it will update. Stage 3 partitions the 16-bit parameters too: each GPU stores only its shard of the weights, and before each layer runs in the forward or backward pass, the GPUs perform an all-gather to temporarily rebuild that layer's full weights, then free them afterwards. For the 7.5B example on 64 GPUs, the paper computes per-GPU model state falling from 120 GB to about 31.4 GB at stage 1, 16.6 GB at stage 2 and 1.88 GB at stage 3.
The idea was introduced by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He of Microsoft in a paper first posted to arXiv in October 2019 and revised in May 2020. It reported training models of over 100 billion parameters on 400 GPUs at about 15 petaflops of throughput, an 8 times increase in trainable model size and up to 10 times better performance than the prior state of the art, and training models of up to 13 billion parameters without any model parallelism. A family of follow-ups came next. ZeRO-Offload (January 2021) moved optimizer state and the optimizer step onto CPU memory, reaching 40 teraflops on one V100 for a 10B model. ZeRO-Infinity (April 2021) added NVMe solid-state storage as a third memory tier and reported over 25 petaflops on 512 V100 GPUs. ZeRO++ (June 2023) attacked communication instead, cutting ZeRO's communication volume by 4 times with quantized collectives. PyTorch documented its own FSDP design in an April 2023 paper reporting performance comparable to standard Distributed Data Parallel on models that DDP cannot hold.
ZeRO is one of several ways to split a model. Tensor parallelism, as in NVIDIA's Megatron-LM, cuts individual matrix multiplications across GPUs, which needs very fast links and changes the model code. Pipeline parallelism assigns different layers to different GPUs and streams micro-batches through them, which introduces idle bubbles and scheduling complexity. ZeRO instead keeps every GPU running the whole model on its own data and only changes where state is stored, so model code usually needs no rewriting. In practice, the largest training runs combine all three, with ZeRO or FSDP handling the data-parallel dimension. The ZeRO paper itself paired its stages with Megatron-style model parallelism to reach its 100-billion-parameter results.
The costs are mostly communication. Stages 1 and 2 move the same amount of data per step as plain data parallelism, which the paper counts as twice the parameter count, because the all-reduce that DDP already performs can be split into a reduce-scatter and an all-gather. Stage 3 raises that to three times the parameter count, a 1.5 times increase, because weights must be gathered once in the forward pass and again in the backward pass. On fast interconnects that overhead can be hidden by overlapping communication with computation, but on slower networks stage 3 can become network-bound. Offloading to CPU or NVMe trades GPU memory for PCIe bandwidth, so it can enable a model that would otherwise not fit while training it noticeably more slowly. Sharding also does nothing for activation memory on its own, so activation checkpointing is usually needed alongside it.
Using ZeRO in practice is mainly configuration. In DeepSpeed, a JSON block named zero_optimization sets the stage and options such as reduce_bucket_size, overlap_comm, and offload_optimizer or offload_param with a device of cpu or nvme. In PyTorch, applying fully_shard to each transformer block and then to the root module gives stage 3 behavior, with reshard_after_forward controlling whether weights are freed after the forward pass. One operational detail catches many users: sharded checkpoints are saved as many per-rank files, so producing a single ordinary weight file requires a consolidation step, such as DeepSpeed's zero_to_fp32.py script, which the docs say needs about twice the final checkpoint size in CPU memory. Choosing a stage is a trade-off: use the lowest stage that fits the model, because each higher stage adds communication.
04工作範例
Estimating per-GPU memory for a 13B model under each ZeRO stage
Take a 13-billion-parameter model trained with mixed-precision Adam, so P = 13e9 and model state is 16 bytes per parameter.
Without ZeRO, every GPU holds 16 x 13e9 = 208 GB, which exceeds an 80 GB GPU.
With stage 1 on N = 8 GPUs: 4P + 12P/8 = 52 GB + 19.5 GB = 71.5 GB per GPU.
With stage 2 on 8 GPUs: 2P + 14P/8 = 26 GB + 22.75 GB = 48.75 GB per GPU.
With stage 3 on 8 GPUs: 16P/8 = 26 GB per GPU.
Add headroom for activations, temporary all-gather buffers and fragmentation, and compare each total to the 80 GB available.
它顯示了什麼
Stage 1 barely fits with almost no room for activations, stage 2 leaves about 31 GB of headroom, and stage 3 leaves about 54 GB at 1.5 times the communication. Check the arithmetic against the formulas in the ZeRO paper and confirm with the framework's memory reports during a short trial run.
現實世界的實施
Microsoft trained Turing-NLG, a 17-billion-parameter language model, with ZeRO, as reported in the ZeRO paper.
DeepSpeed's official ZeRO tutorial shows a 1.5B-parameter GPT-2 that runs out of memory on eight 32 GB V100 GPUs, then fits once ZeRO stage 1 cuts per-GPU optimizer memory from about 18 GB to 2.25 GB.
ZeRO-Offload, published January 2021, trains models of over 13 billion parameters on a single NVIDIA V100 by moving optimizer work to CPU memory, about 10 times what plain PyTorch fits.
PyTorch's fully_shard API (FSDP2) shards every parameter across the data-parallel group and offers a CPUOffloadPolicy, giving PyTorch users ZeRO stage 3 behavior without DeepSpeed.
ZeRO-Infinity extends sharding to CPU and NVMe storage, and its April 2021 paper reports fine-tuning trillion-parameter models on a single NVIDIA DGX-2 node.
ZeRO 和分片優化器的未來
The memory wall that motivated ZeRO keeps widening: the ZeRO-Infinity authors noted in 2021 that the largest dense models had grown over 1,000 times in three years while GPU memory grew about 5 times, from 16 GB to 80 GB. That pressure has shifted research from fitting state toward moving it efficiently. ZeRO++ (June 2023) showed that quantizing the collectives themselves can cut communication volume by 4 times, and PyTorch's per-parameter FSDP2 design reflects a push to make sharding compose cleanly with tensor parallelism, compilers and distributed checkpointing. Open problems include training efficiently over slower or heterogeneous networks, where stage 3's extra gather traffic dominates; keeping offloading to CPU and NVMe from bottlenecking on PCIe bandwidth; and reducing the operational friction of sharded checkpoints. How these trade-offs settle will depend on future interconnect speeds and accelerator memory sizes, which are not known in advance, so teams generally benchmark the lowest workable stage on their own hardware rather than assuming one setting is best.
付諸實踐
為什麼這很重要
- 成本與預算. 多年來,架構決策決定著效能和營運成本。
- 更明確的決策. 技術教育幫助團隊選擇正確的堆疊,而不僅僅是最新的堆疊。
- 品質管控. 更好的工程選擇可以減少生產中的可靠性事故。
留意
- 優化一項基準測試可以隱藏更廣泛的系統弱點。
- 基礎設施和維護成本常常被低估。
- 隨著系統變得更加複雜,安全性和可觀察性差距可能會擴大。
後續步驟
- 在實施之前定義延遲、品質和成本目標。
- 在實際負載和資料條件下進行基準測試。
- 儀器監控錯誤、漂移和使用者影響。
- 在擴展之前準備回滾和事件回應路徑。
資料來源與延伸閱讀
- arXiv (Rajbhandari, Rasley, Ruwase, He, Microsoft)ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- arXiv (Ren et al.)ZeRO-Offload: Democratizing Billion-Scale Model Training
- arXiv (Rajbhandari et al.)ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- arXiv (Wang et al.)ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- arXiv (Zhao et al.)PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- DeepSpeedZero Redundancy Optimizer tutorial
- PyTorch documentationtorch.distributed.fsdp.fully_shard
不斷探索
常見問題
What is ZeRO and Sharded Optimizers?
ZeRO (Zero Redundancy Optimizer) is a memory-saving technique for data-parallel training that splits optimizer states, gradients and eventually the weights themselves across GPUs instead of copying them onto every GPU. Introduced by Microsoft in 2019 and shipped in DeepSpeed, it and its PyTorch counterpart FSDP are how most teams now fit multi-billion-parameter models onto ordinary GPU clusters.
與普通資料並行相比,ZeRO 消除了哪些冗餘?
標準資料並行性在每個 GPU 上儲存優化器狀態、梯度和權重的完整副本; ZeRO 對這些進行分片,因此每個 GPU 僅保留一個切片。
為什麼優化器狀態通常是 Adam 佔用最大記憶體的地方?
Adam 維護每個參數的運行估計,例如第一和第二時刻,與 fp32 主權重相結合可以使模型本身的大小相形見絀。
ZeRO 第 3 階段分片有哪些第 1 期和第 2 期沒有的分片?
階段 1 對優化器狀態進行分片,第 2 階段新增梯度,而第 3 階段則進一步將模型參數分片到 GPU 上。
在 ZeRO Stage 3 中,GPU 如何取得層前向傳遞所需的完整參數?
在計算某個層之前,全收集會在每個 GPU 上組裝其完整參數;一旦完成,非擁有的切片將被釋放以回收記憶體。
哪個 PyTorch 功能本身實現了 ZeRO 風格的分片?
PyTorch 的完全分片資料並行 (FSDP) 對參數、梯度和最佳化器狀態進行分片,動態收集和重新分片,鏡像 ZeRO。