NVIDIA enables multi-GPU inference in Dynamo-Triton 26.07

O que aconteceu

NVIDIA released Dynamo-Triton 26.07, integrating TensorRT multi-device inference to allow single model instances to execute across multiple GPUs using NCCL-backed distributed collectives.

NVIDIA has released Dynamo-Triton 26.07, which enables the multi-device inference capability of the TensorRT backend. This new feature allows a single Triton KIND_MODEL instance to own multiple GPUs, creating per-rank TensorRT execution contexts, CUDA streams, and NCCL communicators. The application interacts with a single named model through a gRPC endpoint, eliminating the need for client-side coordination of GPU ranks.

The release is fully supported starting with TensorRT 11.0. The integration uses Ulysses context parallelism to distribute video tokens across GPUs. In a demonstration using the NVIDIA Cosmos 3 Nano video generation model, the 36-layer denoising transformer was distributed across up to eight GPUs. The distributed graph is compiled into each TensorRT plan before deployment, and the Dynamo-Triton backend loads this versioned plan to expose the multi-rank execution state.

The source provides specific performance metrics for the Cosmos 3 Nano model. On a single GPU, end-to-end latency for a 189-frame video generation was 156.595 seconds. With eight GPUs (CP8), this latency dropped to 34.183 seconds, achieving a 6.09x speedup in transformer RPCs. The transformer RPCs accounted for 93.4% of generation time on a single GPU, falling to 70.2% at CP8, indicating that non-transformer overhead becomes a larger fraction of the total time as GPU count increases.

Detalhes da fonte: developer.nvidia.com ↗

Por que isso importa

This update simplifies the deployment of large generative AI models that exceed single-GPU memory limits. By abstracting multi-GPU coordination into the inference server, developers can reduce request latency without modifying application code, enabling faster video generation and other latency-sensitive workflows.

This development addresses a critical bottleneck in generative AI deployment: the compute and memory demands of large models often exceed the capacity of a single GPU. By integrating multi-device inference directly into the inference server, NVIDIA removes the complexity of distributed coordination from the application layer. This allows teams to trade additional GPU resources for shorter request latency while keeping the application interface stable.

For latency-sensitive workflows such as generative media, this reduction in time-to-result can significantly accelerate user interactions and review cycles. The ability to package the engine as a versioned Triton model and keep rank lifecycle code out of the client simplifies deployment and maintenance for organizations scaling their AI infrastructure.

The practical implication is that developers can now deploy large video generation models with a conventional model-serving interface, rather than building custom distributed inference logic. This lowers the barrier to entry for high-performance generative AI applications that require multi-GPU acceleration.

O que assistir a seguir

Monitor adoption of this feature in enterprise video generation pipelines and whether NVIDIA extends multi-device support to other model architectures or inference frameworks.

Observe how enterprise teams adopt this feature for their own generative AI workloads, particularly in video and image generation where latency is a key performance metric.

Monitor whether NVIDIA extends this multi-device inference capability to other model architectures or inference frameworks beyond TensorRT and Dynamo-Triton.

Watch for third-party benchmarks or independent evaluations of the performance and cost-effectiveness of multi-GPU inference compared to single-GPU or cloud-based alternatives.

Guias e questionários relacionados