NVIDIA stated it has achieved a report giant language mannequin (LLM) inference pace, saying that an NVIDIA DGX B200 node with eight NVIDIA Blackwell GPUs achieved greater than 1,000 tokens per second (TPS) per person on the 400-billion-parameter Llama 4 Maverick mannequin.
NVIDIA stated the mannequin is the biggest and strongest within the Llama 4 assortment and that the pace was independently measured by the AI benchmarking service Artificial Analysis.
NVIDIA added that Blackwell reaches 72,000 TPS/server at their highest throughput configuration.
The corporate stated it made software program optimizations utilizing TensorRT-LLM and educated a speculative decoding draft mannequin utilizing EAGLE-3 techniques. Combining these approaches, NVIDIA has achieved a 4x speed-up relative to the perfect prior Blackwell baseline, NVIDIA stated.
“The optimizations described under considerably improve efficiency whereas preserving response accuracy,” NVIDIA stated in a weblog posted yesterday. “We leveraged FP8 information varieties for GEMMs, Combination of Consultants (MoE), and Consideration operations to scale back the mannequin dimension and make use of the excessive FP8 throughput attainable with Blackwell Tensor Core technology. Accuracy when utilizing the FP8 information format matches that of Artificial Analysis BF16 across many metrics….”Most generative AI software contexts require a steadiness of throughput and latency, making certain that many purchasers can concurrently take pleasure in a “adequate” expertise. Nonetheless, for important purposes that should make necessary choices at pace, minimizing latency for a single consumer turns into paramount. Because the TPS/person report exhibits, Blackwell {hardware} is your best option for any activity—whether or not you could maximize throughput, steadiness throughput and latency, or decrease latency for a single person (the main target of this publish).
Beneath is an outline of the kernel optimizations and fusions (denoted in red-dashed squares) NVIDIA utilized throughout the inference. NVIDIA applied a number of low-latency GEMM kernels, and utilized varied kernel fusions (like FC13 + SwiGLU, FC_QKV + attn_scaling and AllReduce + RMSnorm) to ensure Blackwell excels on the minimal latency situation.
Overview of the kernel optimizations & fusions used for Llama 4 Maverick
NVIDIA optimized the CUDA kernels for GEMMs, MoE, and Consideration operations to realize the perfect efficiency on the Blackwell GPUs.
- Utilized spatial partitioning (also called warp specialization) and designed the GEMM kernels to load information from reminiscence in an environment friendly method to maximise utilization of the large reminiscence bandwidth that the NVIDIA DGX system affords—64TB/s HBM3e bandwidth in whole.
- Shuffled the GEMM weight in a swizzled format to permit higher structure when loading the computation end result from Tensor Memory after the matrix multiplication computations utilizing Blackwell’s fifth-generation Tensor Cores.
- Optimized the efficiency of the eye kernels by dividing the computations alongside the sequence size dimension of the Ok and V tensors, permitting computations to run in parallel throughout a number of CUDA thread blocks. As well as, NVIDIA utilized distributed shared memory to effectively scale back outcomes throughout the thread blocks in the identical thread block cluster with out the necessity to entry the worldwide reminiscence.
The rest of the weblog could be found here.
