概述
Offloaded bulk data movement can reduce CPU copying and kernel-path overhead; CPUs and software still configure, initiate and manage the communication.
深入探讨
Distributed AI training can spend substantial time moving gradients and other tensors between devices and servers. InfiniBand provides a switched fabric with RDMA support and link-level flow control. NDR denotes a 400 Gb/s link mode in the cited NVIDIA hardware documentation; application throughput and latency depend on messages, adapters, topology and workload. RDMA-capable adapters can move bulk data to registered buffers without the repeated host copying of a conventional path. This does not mean software or CPUs have no role: applications create communication objects, register buffers and post work. RoCE carries RDMA over Ethernet, with its own network-design requirements. Select and benchmark a fabric using the actual collective traffic, congestion and failure conditions.
现实世界的实施
在人工智能超级计算机中连接数千个 GPU，以便梯度数据在分布式训练期间以微秒的速度在节点之间移动
让一台服务器直接读取另一台服务器的内存 (RDMA) 以加速分布式文件系统和数据库，而无需 CPU 开销
通过 InfiniBand 运行 NCCL all-reduce 操作以跨 GPU 集群同步模型权重
使用 RoCE 将 RDMA 式低延迟传输引入现有以太网数据中心网络
InfiniBand 和 RDMA 网络的未来
Interconnects and adapters continue to evolve, and vendors offer different bandwidth modes and offloads. Teams should compare measured throughput, latency, CPU work, congestion handling and reliability under their own workload. In-network reductions and closer accelerator integration may help particular configurations, but a headline link rate does not determine training performance by itself. Keep hardware, firmware and software versions with benchmark results.
付诸实践
为什么这很重要
- 成本与预算. 多年来，架构决策决定着性能和运营成本。
- 更清晰的判决. 技术教育帮助团队选择正确的堆栈，而不仅仅是最新的堆栈。
- 质量控制. 更好的工程选择可以减少生产中的可靠性事故。
留意
- 优化一项基准测试可以隐藏更广泛的系统弱点。
- 基础设施和维护成本常常被低估。
- 随着系统变得更加复杂，安全性和可观察性差距可能会扩大。
后续步骤
- 在实施之前定义延迟、质量和成本目标。
- 在实际负载和数据条件下进行基准测试。
- 仪器监控错误、漂移和用户影响。
- 在扩展之前准备回滚和事件响应路径。
不断探索
常见问题
什么是 InfiniBand 和 RDMA 网络？
InfiniBand is a high-speed switched interconnect used in clusters, and RDMA supports direct transfers between registered memory buffers. Offloaded bulk data movement can reduce CPU copying and kernel-path overhead; CPUs and software still configure, initiate and manage the communication.
What does an RDMA-capable bulk data path aim to reduce?
RDMA offloads data movement but software still sets up and manages communication.
How can an RDMA-capable InfiniBand data path reduce bulk-transfer overhead?
RDMA hardware can move data between prepared buffers with less repeated host copying and kernel-path work. Software still handles setup, registration, permissions, and completion processing.
NDR InfiniBand 提供的每链路带宽大致是多少？
A four-lane NDR400 InfiniBand link is rated at 400 Gb/s. This link-rate figure does not guarantee an application’s effective transfer throughput.
在 RDMA 中，“队列对”有什么用？
应用程序将工作请求发布到发送和接收队列（队列对）；主机通道适配器读取它们并执行直接内存传输。
什么是RoCE？
RoCE 代表融合以太网上的 RDMA，允许在标准以太网结构（而不是本机 InfiniBand）上进行低延迟、内核旁路传输。