NVIDIA’s TensorRT Edge-LLM completed the 1,007-turn MLPerf Inference v6.1 Edge Agentic performance workload on one Jetson AGX Thor Developer Kit in 24 minutes and 36 seconds. That was 6.4x faster than a published llama.cpp reference run lasting 2 hours and 37 minutes, giving developers a measured example of running long, tool-using agent workflows on an edge system.
The NVIDIA submission ran Qwen3.6-27B in SingleStream mode on a Jetson AGX Thor Developer Kit with 128 GB of unified memory and the MAXN power mode. NVIDIA reported output throughput of 52.33 tokens per second, a median time to first token of 247.12 milliseconds and a median time per output token of 14.68 milliseconds. Its overall accuracy on the Berkeley Function Calling Leaderboard v4 evaluation was 87.94%.
What the benchmark measures
MLPerf Edge Agentic evaluates an OpenAI-compatible model endpoint through separate performance and accuracy phases. The performance workload replays 20 recorded software-engineering agent conversations: the model issues a tool call, receives the result and continues within the accumulated history. Across 1,007 generated turns, input length rises to about 23,500 tokens. Inline accuracy based on Intersection of Union is checked during this phase to verify that the agent is operating correctly.
The accuracy phase uses 995 single-turn BFCL v4 prompts with reasoning disabled. It tests whether the model chooses the correct function, supplies valid arguments and refrains from calling a tool when none is required. This separation matters because the reported 6.4x figure concerns end-to-end completion time for the multi-turn performance workload, while 87.94% is the separately reported BFCL accuracy result.
Three optimizations behind the result
The submission used NVFP4 for model weights and activations, including the language-model head, while storing the KV cache in FP8. NVFP4 is a 4-bit floating-point format supported by the Blackwell GPU in Jetson AGX Thor. NVIDIA says the smaller representation reduces memory traffic during low-batch decoding and leaves more of the system’s unified memory available for long context, speculative-decoding state and other application workloads. A calibrated NVFP4 checkpoint is available for the evaluated model.
TensorRT Edge-LLM also avoids repeatedly processing the shared history that appears in successive agent turns. It detects reusable prompt prefixes and restores cached attention KV pages. Because Qwen3.6 has a hybrid architecture, the runtime additionally restores recurrent state and partial KV-page state, then processes only the new suffix. NVIDIA reports that approximately 96% of prompt tokens in this workload were served from a hot cache: only about 0.5 million of 13.6 million total prompt tokens were prefetched again.
For generation, the runtime uses tree-based multi-token prediction. A draft model proposes several continuations in a tree, allowing the target model to verify multiple candidates in one forward pass and accept the matching path. The benchmark configuration used eight draft steps, the top two candidates at each depth and a 16-node verification tree. NVIDIA says this configuration could deliver about 40% more decoding performance on the workload than linear multi-token prediction with three draft steps. That figure is a vendor-reported comparison specific to this workload, not the overall 6.4x result.
Reproduction details and comparison limits
The implementation is published on the release/0.9.1-mlpinf branch, which contains the export settings, TensorRT engine build commands, server configuration and MLPerf client configuration. Reproducing the submission requires the calibrated checkpoint, base and draft TensorRT engines, the OpenAI-compatible TensorRT Edge-LLM server and the MLCommons endpoint harness. The supplied configuration runs with temperature zero, seed 42, reasoning disabled and concurrency one.
The comparison is not an isolated test of runtime software: the TensorRT submission used NVFP4 weights and activations with an FP8 KV cache, while the llama.cpp reference used Q4_K_M quantization. The reported outcome therefore reflects the complete submitted configurations, including their quantization and inference techniques, on Jetson AGX Thor.
Source: NVIDIA Developer Blog.
Definition. TensorRT Edge-LLM is the runtime used in NVIDIA’s Jetson AGX Thor submission to process long, tool-using agent conversations through an OpenAI-compatible model endpoint.
| TensorRT Edge-LLM submission | Published llama.cpp reference |
|---|---|
| 24 minutes 36 seconds | 2 hours 37 minutes |
| NVFP4 weights and activations | Q4_K_M quantization |
| FP8 KV cache | KV cache format not stated |
| 6.4x reported completion-time advantage | Reference baseline |
Key takeaways
- The submission achieved 52.33 output tokens per second, a 247.12-millisecond median time to first token and a 14.68-millisecond median time per output token.
- Its separately measured BFCL v4 accuracy was 87.94%.
- Approximately 96% of prompt tokens were served from a hot cache, with about 0.5 million of 13.6 million prompt tokens prefetched again.
- The runtime combined NVFP4 weights and activations, FP8 KV cache storage, reusable prompt-prefix caching and tree-based multi-token prediction.
- The 6.4x result compares complete configurations: TensorRT used NVFP4 and an FP8 KV cache, while the llama.cpp reference used Q4_K_M quantization.
FAQ
How long did TensorRT Edge-LLM take to complete the workload?
It completed the 1,007-turn MLPerf Inference v6.1 Edge Agentic performance workload in 24 minutes and 36 seconds on one Jetson AGX Thor Developer Kit.
What accuracy did the NVIDIA submission report?
It reported 87.94% overall accuracy on the separate Berkeley Function Calling Leaderboard v4 evaluation.
What optimizations contributed to the result?
The submission used NVFP4 weights and activations, an FP8 KV cache, reusable prompt-prefix caching and tree-based multi-token prediction.
Is the 6.4x figure a runtime-only comparison?
No. It compares complete submitted configurations using different quantization formats and inference techniques on Jetson AGX Thor.
How can the submission be reproduced?
The published release/0.9.1-mlpinf branch includes export settings, engine build commands, server configuration and MLPerf client configuration; reproduction also requires the calibrated checkpoint, base and draft engines, server and MLCommons endpoint harness.
