Research group Epoch AI released its Inference Performance Index on November 9, 2026. The new benchmark evaluates AI chips based on their ability to serve users simultaneously while maintaining fixed output speeds.
The index incorporates compute capacity, memory bandwidth, and inter-chip communication. It indicates that the Nvidia GB300, based on Blackwell Ultra architecture, outperforms the H100 by a factor of 10 in inference tasks.
Technical specifications for the GB300 include 288 GB of HBM3e memory per GPU and approximately 8 TB/s of memory bandwidth, compared to roughly 3.35 TB/s for the H100.
The GB300 is typically deployed as part of the GB300 NVL72 rack system, which connects 72 GPUs. This configuration supports up to 20–40 TB of pooled memory and utilizes upgraded NVLink interconnects.
Independent data and MLPerf tests suggest the GB300 NVL72 offers 5–20 times improvements in throughput, cost-per-token, and concurrency over Hopper-generation platforms depending on the model used.
Nvidia reports that the GB300 NVL72 achieves up to 10 times more tokens per second per user and 5 times more tokens per second per megawatt compared to Hopper-based systems at the rack scale.
