概述
A resumable training checkpoint also records the state needed to continue an interrupted run, and restart tests establish what it can restore.
深入探讨
A weight checkpoint contains model parameters; a resumable training checkpoint also needs relevant optimizer, scheduler, step, random-number-generator, and data-loading state. Sharding divides large tensors or state across files and workers so saving and loading can avoid gathering everything on one rank. PyTorch Distributed Checkpoint supports distributed state saving and resharding during loading when supported by the chosen state and configuration. Sharded model weights on the Hugging Face Hub can use an index mapping parameter names to files, but weight shards alone do not capture a training run’s full progress. Applications must include their required state and validate recovery after an interrupted save or a changed environment.
现实世界的实施
前沿模型运行在数千个 GPU 上，每隔几百步自动保存分片检查点，因此单个失败的节点只需要几分钟，而不是几天。
Hugging Face 将大型开放模型分发为多个 safetensors 分片和一个 index.json，以便用户可以逐个下载和加载它。
研究人员恢复中断的微调，恢复精确的优化器动力、步数和数据加载器位置以无缝继续。
在廉价的可抢占式云 GPU 上进行现货实例训练，其中频繁的分片检查点使作业能够在被驱逐和重新安排的情况下幸存下来。
检查点分片和可恢复训练的未来
检查点正在从周期性的停止世界事件转变为异步且几乎免费的事件。预计会有更多的内存中和重叠检查点，在训练继续的同时在后台写入分片，加上纠删码和复制检查点，可以在千个 GPU 规模上常见的节点故障中幸存下来。云对象存储和更快的本地 NVMe 层将托管分片，安全张量等标准化格式将不断改进训练恢复和推理部署的安全、快速、部分加载。
付诸实践
为什么这很重要
- 成本与预算. 多年来，架构决策决定着性能和运营成本。
- 更清晰的判决. 技术教育帮助团队选择正确的堆栈，而不仅仅是最新的堆栈。
- 质量控制. 更好的工程选择可以减少生产中的可靠性事故。
留意
- 优化一项基准测试可以隐藏更广泛的系统弱点。
- 基础设施和维护成本常常被低估。
- 随着系统变得更加复杂，安全性和可观察性差距可能会扩大。
后续步骤
- 在实施之前定义延迟、质量和成本目标。
- 在实际负载和数据条件下进行基准测试。
- 仪器监控错误、漂移和用户影响。
- 在扩展之前准备回滚和事件响应路径。
不断探索
常见问题
什么是检查点分片和可恢复训练？
Checkpoint sharding saves a large model or training state across multiple pieces for manageable storage and loading. A resumable training checkpoint also records the state needed to continue an interrupted run, and restart tests establish what it can restore.
为什么大模型检查点要分成分片？
将巨大的检查点（数百 GB）作为一个文件是不切实际的；分片允许许多队列并行写入其切片并启用分段加载。
Which extra state is typically needed for training resumption beyond model weights?
Weight shards alone do not restore optimizer history or the next intended batch. The application must save the required training progress and randomness state.
长期工作可恢复培训的主要好处是什么？
通过可恢复的检查点，中断的运行会重新加载其上次保存的状态并继续，而不是丢弃数天的计算。
In a sharded distributed checkpoint, how can ranks avoid gathering the entire state on one worker?
Distributed checkpointing uses local state partitions and metadata. Ordinary replicated data parallelism does not automatically shard every tensor.
索引文件在分片检查点中起什么作用？
索引（例如，index.json）记录哪个分片保存每个参数，以便加载器可以获取并重新组装正确的片段。