Technische GIDS
ZeRO en Sharded Optimizers
ZeRO (Zero Redundancy Optimizer) is a memory-saving technique for data-parallel training that splits optimizer states, gradients and eventually the weights themselves across GPUs instead of copying them onto every GPU.
Luister naar deze gidsAI-verteld · ~8 min
Op deze pagina9 minuten lezen
Overzicht
Introduced by Microsoft in 2019 and shipped in DeepSpeed, it and its PyTorch counterpart FSDP are how most teams now fit multi-billion-parameter models onto ordinary GPU clusters.
Belangrijkste inzichten
- Mixed-precision Adam needs about 16 bytes per parameter, so a 7.5-billion-parameter model needs roughly 120 GB of model state before activations.
- ZeRO stage 1 shards optimizer state, stage 2 also shards gradients, and stage 3 also shards the model weights themselves.
- Stages 1 and 2 cost the same communication as ordinary data parallelism, while stage 3 costs about 1.5 times as much.
- The 2019 ZeRO paper from Microsoft reported training models over 100 billion parameters on 400 GPUs at about 15 petaflops.
- PyTorch FSDP provides ZeRO-style full sharding natively, and ZeRO-Offload and ZeRO-Infinity extend it to CPU and NVMe memory.
Diepe duik
ZeRO, the Zero Redundancy Optimizer, is a way to train neural networks that are too big for one GPU's memory while keeping the simple programming model of data parallelism. In ordinary data parallelism, every GPU holds a full copy of the model, processes a different slice of the training batch, and the GPUs average their gradients after each step. That is easy to use but wasteful: with 64 GPUs, there are 64 identical copies of everything. ZeRO removes that duplication by giving each GPU responsibility for only a fraction of the training state, and fetching the rest from its peers just when it is needed. A sharded optimizer is the general name for this pattern; PyTorch's Fully Sharded Data Parallel (FSDP) is the best-known implementation outside Microsoft's DeepSpeed library.
The reason duplication hurts is that the weights are not the biggest item. The ZeRO paper does the accounting for mixed-precision training with the Adam optimizer, where each parameter needs 2 bytes for a 16-bit copy of the weight, 2 bytes for its 16-bit gradient, and 12 bytes of 32-bit optimizer state: a master copy of the weight plus Adam's two running averages, momentum and variance. That is 16 bytes per parameter, so a 7.5-billion-parameter model needs about 120 GB just for model state, before counting activations, the intermediate values saved for the backward pass. The optimizer state alone is three quarters of the total, which is why sharding it first gives the biggest win.
ZeRO works in three cumulative stages. Stage 1 partitions the optimizer states: each of N GPUs keeps Adam's state for only 1/N of the parameters and updates only that slice, after which the updated weights are shared with everyone. Stage 2 also partitions the gradients: instead of every GPU receiving the full averaged gradient, a reduce-scatter operation sums gradients across GPUs and delivers each GPU only the slice it will update. Stage 3 partitions the 16-bit parameters too: each GPU stores only its shard of the weights, and before each layer runs in the forward or backward pass, the GPUs perform an all-gather to temporarily rebuild that layer's full weights, then free them afterwards. For the 7.5B example on 64 GPUs, the paper computes per-GPU model state falling from 120 GB to about 31.4 GB at stage 1, 16.6 GB at stage 2 and 1.88 GB at stage 3.
The idea was introduced by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase and Yuxiong He of Microsoft in a paper first posted to arXiv in October 2019 and revised in May 2020. It reported training models of over 100 billion parameters on 400 GPUs at about 15 petaflops of throughput, an 8 times increase in trainable model size and up to 10 times better performance than the prior state of the art, and training models of up to 13 billion parameters without any model parallelism. A family of follow-ups came next. ZeRO-Offload (January 2021) moved optimizer state and the optimizer step onto CPU memory, reaching 40 teraflops on one V100 for a 10B model. ZeRO-Infinity (April 2021) added NVMe solid-state storage as a third memory tier and reported over 25 petaflops on 512 V100 GPUs. ZeRO++ (June 2023) attacked communication instead, cutting ZeRO's communication volume by 4 times with quantized collectives. PyTorch documented its own FSDP design in an April 2023 paper reporting performance comparable to standard Distributed Data Parallel on models that DDP cannot hold.
ZeRO is one of several ways to split a model. Tensor parallelism, as in NVIDIA's Megatron-LM, cuts individual matrix multiplications across GPUs, which needs very fast links and changes the model code. Pipeline parallelism assigns different layers to different GPUs and streams micro-batches through them, which introduces idle bubbles and scheduling complexity. ZeRO instead keeps every GPU running the whole model on its own data and only changes where state is stored, so model code usually needs no rewriting. In practice, the largest training runs combine all three, with ZeRO or FSDP handling the data-parallel dimension. The ZeRO paper itself paired its stages with Megatron-style model parallelism to reach its 100-billion-parameter results.
The costs are mostly communication. Stages 1 and 2 move the same amount of data per step as plain data parallelism, which the paper counts as twice the parameter count, because the all-reduce that DDP already performs can be split into a reduce-scatter and an all-gather. Stage 3 raises that to three times the parameter count, a 1.5 times increase, because weights must be gathered once in the forward pass and again in the backward pass. On fast interconnects that overhead can be hidden by overlapping communication with computation, but on slower networks stage 3 can become network-bound. Offloading to CPU or NVMe trades GPU memory for PCIe bandwidth, so it can enable a model that would otherwise not fit while training it noticeably more slowly. Sharding also does nothing for activation memory on its own, so activation checkpointing is usually needed alongside it.
Using ZeRO in practice is mainly configuration. In DeepSpeed, a JSON block named zero_optimization sets the stage and options such as reduce_bucket_size, overlap_comm, and offload_optimizer or offload_param with a device of cpu or nvme. In PyTorch, applying fully_shard to each transformer block and then to the root module gives stage 3 behavior, with reshard_after_forward controlling whether weights are freed after the forward pass. One operational detail catches many users: sharded checkpoints are saved as many per-rank files, so producing a single ordinary weight file requires a consolidation step, such as DeepSpeed's zero_to_fp32.py script, which the docs say needs about twice the final checkpoint size in CPU memory. Choosing a stage is a trade-off: use the lowest stage that fits the model, because each higher stage adds communication.
04Uitgewerkt voorbeeld
Estimating per-GPU memory for a 13B model under each ZeRO stage
Take a 13-billion-parameter model trained with mixed-precision Adam, so P = 13e9 and model state is 16 bytes per parameter.
Without ZeRO, every GPU holds 16 x 13e9 = 208 GB, which exceeds an 80 GB GPU.
With stage 1 on N = 8 GPUs: 4P + 12P/8 = 52 GB + 19.5 GB = 71.5 GB per GPU.
With stage 2 on 8 GPUs: 2P + 14P/8 = 26 GB + 22.75 GB = 48.75 GB per GPU.
With stage 3 on 8 GPUs: 16P/8 = 26 GB per GPU.
Add headroom for activations, temporary all-gather buffers and fragmentation, and compare each total to the 80 GB available.
Wat het laat zien
Stage 1 barely fits with almost no room for activations, stage 2 leaves about 31 GB of headroom, and stage 3 leaves about 54 GB at 1.5 times the communication. Check the arithmetic against the formulas in the ZeRO paper and confirm with the framework's memory reports during a short trial run.
Implementatie in de echte wereld
Microsoft trained Turing-NLG, a 17-billion-parameter language model, with ZeRO, as reported in the ZeRO paper.
DeepSpeed's official ZeRO tutorial shows a 1.5B-parameter GPT-2 that runs out of memory on eight 32 GB V100 GPUs, then fits once ZeRO stage 1 cuts per-GPU optimizer memory from about 18 GB to 2.25 GB.
ZeRO-Offload, published January 2021, trains models of over 13 billion parameters on a single NVIDIA V100 by moving optimizer work to CPU memory, about 10 times what plain PyTorch fits.
PyTorch's fully_shard API (FSDP2) shards every parameter across the data-parallel group and offers a CPUOffloadPolicy, giving PyTorch users ZeRO stage 3 behavior without DeepSpeed.
ZeRO-Infinity extends sharding to CPU and NVMe storage, and its April 2021 paper reports fine-tuning trillion-parameter models on a single NVIDIA DGX-2 node.
De toekomst van ZeRO en Sharded Optimizers
The memory wall that motivated ZeRO keeps widening: the ZeRO-Infinity authors noted in 2021 that the largest dense models had grown over 1,000 times in three years while GPU memory grew about 5 times, from 16 GB to 80 GB. That pressure has shifted research from fitting state toward moving it efficiently. ZeRO++ (June 2023) showed that quantizing the collectives themselves can cut communication volume by 4 times, and PyTorch's per-parameter FSDP2 design reflects a push to make sharding compose cleanly with tensor parallelism, compilers and distributed checkpointing. Open problems include training efficiently over slower or heterogeneous networks, where stage 3's extra gather traffic dominates; keeping offloading to CPU and NVMe from bottlenecking on PCIe bandwidth; and reducing the operational friction of sharded checkpoints. How these trade-offs settle will depend on future interconnect speeds and accelerator memory sizes, which are not known in advance, so teams generally benchmark the lowest workable stage on their own hardware rather than assuming one setting is best.
Het in de praktijk brengen
Waarom het ertoe doet
- Kosten en budget. Architectuurbeslissingen bepalen jarenlang de prestaties en bedrijfskosten.
- Duidelijkere beslissingen. Technisch onderwijs helpt teams bij het kiezen van de juiste stapel, niet alleen de nieuwste.
- Kwaliteitscontrole. Betere technische keuzes verminderen het aantal betrouwbaarheidsincidenten in de productie.
Pas op voor
- Het optimaliseren van één benchmark kan bredere systeemzwakheden verbergen.
- Infrastructuur- en onderhoudskosten worden vaak onderschat.
- De lacunes op het gebied van beveiliging en waarneembaarheid kunnen groter worden naarmate systemen complexer worden.
Volgende stappen
- Definieer latentie-, kwaliteits- en kostendoelen vóór implementatie.
- Benchmark onder realistische belasting- en gegevensomstandigheden.
- Instrumentbewaking op fouten, drift en gebruikersimpact.
- Bereid rollback- en incidentresponspaden voor voordat u gaat schalen.
Bronnen en verdere literatuur
- arXiv (Rajbhandari, Rasley, Ruwase, He, Microsoft)ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- arXiv (Ren et al.)ZeRO-Offload: Democratizing Billion-Scale Model Training
- arXiv (Rajbhandari et al.)ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
- arXiv (Wang et al.)ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- arXiv (Zhao et al.)PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- DeepSpeedZero Redundancy Optimizer tutorial
- PyTorch documentationtorch.distributed.fsdp.fully_shard
Blijf verkennen
Gratis nieuwsbrief
Ontvang de dagelijkse AI-briefing
Drie verifieerde AI-verhalen elke weekdagmorgen, geschreven in eenvoudig Engels. Gratis voor altijd, geen advertenties.
Eén e-mail elke weekdag. Afmelden met één klik. We verkopen of delen je adres nooit.
Test jezelf
Doe de ZeRO- en Sharded Optimizers-quiz
Directe feedback op elke antwoord, en een deelbare certificaat met een verifieerbare ID zodra je een cursus hebt afgerond.
Ondersteun gratis AI-onderwijs. AI Understanding is een 501(c)(3) non-profit — geen advertenties, geen betaalmuur, nooit. Doe een donatie
Veelgestelde vragen
Wat zijn ZeRO en Sharded Optimizers?
ZeRO (Zero Redundancy Optimizer) is a memory-saving technique for data-parallel training that splits optimizer states, gradients and eventually the weights themselves across GPUs instead of copying them onto every GPU. Introduced by Microsoft in 2019 and shipped in DeepSpeed, it and its PyTorch counterpart FSDP are how most teams now fit multi-billion-parameter models onto ordinary GPU clusters.
Welke redundantie elimineert ZeRO vergeleken met gewoon dataparallellisme?
Standaard gegevensparallellisme slaat een volledige kopie op van de optimalisatiestatus, gradiënten en gewichten op elke GPU; ZeRO scherft deze zodat elke GPU slechts een segment bevat.
Waarom is de optimalisatiestatus vaak het grootste geheugenzwijn bij Adam?
Adam houdt lopende schattingen bij, zoals het eerste en tweede moment per parameter, die in combinatie met fp32-mastergewichten de eigen grootte van het model kunnen verkleinen.
Wat doet ZeRO Stage 3 met wat Stage 1 en 2 niet doen?
Fase 1 vernietigt de optimalisatiestatus, Fase 2 voegt gradiënten toe en Fase 3 gaat nog verder door de modelparameters ook over GPU's te verdelen.
Hoe krijgt een GPU in ZeRO Stage 3 de volledige parameters die hij nodig heeft voor de voorwaartse doorgang van een laag?
Voordat een laag wordt berekend, verzamelt een all-gather de volledige parameters op elke GPU; Zodra dit is gedaan, worden de niet-eigen segmenten vrijgegeven om geheugen terug te winnen.
Welke PyTorch-functie implementeert native sharding in ZeRO-stijl?
PyTorch's Fully Sharded Data Parallel (FSDP) scherft parameters, gradiënten en optimalisatiestatussen, verzamelt ze en herhardt ze ter plekke, in navolging van ZeRO.
Blijf leren
Gerelateerde gidsen
Er zijn meer handleidingen voor dit onderwerp geselecteerd