NVIDIA has enabled TensorRT multi-device inference in Dynamo-Triton 26.07, allowing applications to invoke a distributed TensorRT model through one gRPC endpoint instead of managing GPU ranks themselves. For inference teams, that provides a conventional serving interface while letting them assign more GPUs to a request to reduce latency.
The capability, fully supported from TensorRT 11.0, runs one TensorRT network across multiple GPUs with NCCL-backed distributed collectives while retaining TensorRT inference optimizations. A single KIND_MODEL instance can own the participating GPUs and create the per-rank execution contexts, CUDA streams and NCCL communicators needed to launch them together.
Serving a precompiled distributed plan
Dynamo-Triton does not convert an ordinary single-device engine into a distributed one. The distributed graph must already be compiled into each context-parallel TensorRT plan; Dynamo-Triton then loads the versioned plan, creates its multi-rank execution state and exposes the model endpoint. Its configuration enables the multi-device backend path and identifies the GPUs assigned to the ranks.
NVIDIA demonstrated the integration with Cosmos 3 Nano video generation. Diffusers continued to handle prompts, latents, classifier-free guidance, scheduling, VAE decoding and frame postprocessing, while Dynamo-Triton served the model’s 36-layer denoising transformer. That transformer represented 93.4% of the single-GPU generation time, making it the principal target for acceleration.
The workflow performed 35 denoising steps. Because each step required an unconditional or negative prediction and a prompt-conditioned prediction, the Diffusers proxy made two sequential Triton requests per step, totaling 70 transformer RPCs for each generated video.
How the workload was divided
TensorRT used Ulysses context parallelism to distribute 44,160 video tokens across as many as eight GPUs. At context-parallel size eight, each rank processed 5,520 video tokens outside attention, while the shorter 2,992-token text path remained replicated. Within every transformer layer, Ulysses changed the partitioning axis around attention so each rank handled the full video sequence for a nonoverlapping subset of attention heads.
The engine was exported from PyTorch and compiled with Torch-TensorRT. NVIDIA used converters that mapped reduce-scatter, all-to-all and all-gather operations to TensorRT’s public distributed-collective layer. Each accepted plan contained two initial reduce-scatters, three all-to-alls in each of the 36 layers, and a final all-gather—for a total of two reduce-scatters, 108 all-to-alls and one all-gather.
Latency results and methodology
NVIDIA compared single-GPU, two-GPU, four-GPU and eight-GPU variants on the same healthy eight-GPU system. Every run produced 189 frames at 1280×720 and 24 FPS using 35 denoising steps. Each result followed one warm-up with five measured complete generations. Timing included prompt processing, all 70 Dynamo-Triton calls, classifier-free-guidance and scheduler updates, VAE decoding and frame postprocessing; model loading and MP4 encoding were excluded.
Mean end-to-end latency fell from 156.595 seconds on one GPU to 87.999 seconds on two, 53.093 seconds on four and 34.183 seconds on eight. Those results corresponded to speedups of 1.78x, 2.95x and 4.58x. Mean transformer RPC time declined from 146.192 seconds to 77.548, 42.661 and 23.993 seconds, with the eight-GPU configuration reaching a 6.09x RPC speedup.
Scaling was not linear. Work outside the measured RPC path remained between 10.2 and 10.5 seconds, so its share became more prominent as transformer execution accelerated. Transformer RPCs accounted for 93.4% of the single-GPU total but 70.2% with eight GPUs.
Output checks and deployment trade-offs
NVIDIA used the same seed and generation profile for every variant, sampled frames 0, 47, 94, 141 and 188, and checked format, temporal variation and similarity to the single-device output. All multi-GPU variants passed the configured thresholds of mean absolute error no greater than 25 and peak signal-to-noise ratio of at least 18 dB. CP2 and CP4 recorded MAE 12.759 and PSNR 21.111 dB; CP8 recorded MAE 16.316 and PSNR 19.400 dB.
The outputs were not claimed to be pixel-identical. More broadly, the results describe a resource-for-latency trade-off for a single generation, not a complete deployment assessment: the benchmark did not measure concurrent-request throughput, cost per video or total cost of ownership. Teams would therefore need to compare the latency improvement and additional GPU allocation with their own service objectives and economics.
Source: NVIDIA Developer Blog
Definition. Dynamo-Triton multi-device inference is a serving capability that runs a precompiled distributed TensorRT network across multiple GPUs behind one model endpoint.
| GPU configuration | Mean end-to-end latency |
|---|---|
| 1 GPU | 156.595 seconds |
| 2 GPUs | 87.999 seconds |
| 4 GPUs | 53.093 seconds |
| 8 GPUs | 34.183 seconds |
Key takeaways
- One KIND_MODEL instance can manage participating GPUs, per-rank execution contexts, CUDA streams and NCCL communicators.
- The feature requires a graph already compiled into a context-parallel TensorRT plan; it does not distribute an ordinary single-device engine automatically.
- Mean end-to-end latency declined from 156.595 seconds on one GPU to 34.183 seconds on eight GPUs.
- Eight GPUs produced a 6.09x transformer RPC speedup, while end-to-end speedup reached 4.58x because work outside the RPC path remained.
- All tested multi-GPU outputs passed the configured MAE and PSNR thresholds, though NVIDIA did not claim pixel-identical results.
- The benchmark describes a resource-for-latency trade-off and does not establish throughput or deployment economics.
FAQ
What does Dynamo-Triton 26.07 add?
It enables applications to invoke a distributed TensorRT model across multiple GPUs through one gRPC endpoint.
Can Dynamo-Triton convert a single-GPU TensorRT engine into a distributed engine?
No. The distributed graph must already be compiled into each context-parallel TensorRT plan.
How much did eight GPUs reduce Cosmos 3 Nano generation latency?
Mean end-to-end latency fell from 156.595 seconds on one GPU to 34.183 seconds on eight GPUs, a 4.58x speedup.
How was the benchmark measured?
Each configuration used one warm-up followed by five measured complete generations of 189 frames at 1280×720 and 24 FPS with 35 denoising steps.
Were the multi-GPU outputs pixel-identical?
No pixel-identical claim was made, although every multi-GPU variant passed the configured MAE and PSNR thresholds.
Did the test measure deployment economics or throughput?
No. It did not measure concurrent-request throughput, cost per video or total cost of ownership.
