GPU Benchmark Comparison: What the Numbers Actually Tell You

The first thing most teams do when evaluating hardware is pull up a gpu benchmark comparison chart and start sorting by teraflops. It is an understandable instinct, but it leads to poor decisions more often than not. Published benchmarks from hardware vendors are designed to showcase peak theoretical performance under ideal conditions that rarely match real workloads. An A100 advertises 312 teraflops of FP16 throughput. An H100 SXM advertises 989 teraflops of FP16 with sparsity. Those numbers are accurate in the sense that the silicon can sustain them on carefully constructed matrix multiplication kernels that perfectly align with the tensor core architecture. They tell you almost nothing about what your specific model, at your specific batch size, with your specific data pipeline, will achieve in practice. A gpu benchmark comparison that relies on vendor spec sheets is not a comparison at all. It is marketing.

The gap between synthetic benchmarks and real workload performance is widest in training. MLPerf results, the closest thing the industry has to a standardized gpu benchmark comparison, measure time to convergence on specific models with specific hyperparameters at specific scales. Those results are useful for comparing hardware generations, but they hide critical variables. The teams that submit MLPerf results spend weeks tuning NCCL parameters, kernel launch configurations, data loader prefetch depths, and gradient accumulation schedules. Your team will not do that. Your team will use default PyTorch settings, a standard data loader, and whatever NCCL version shipped with your container image. Under those conditions, most teams extract 40 to 60 percent of the theoretical peak throughput from their GPUs. When comparing a100 vs h100 performance for a real training workload, the practical speedup is often 1.8 to 2.2x rather than the 3x that the spec sheets imply. That is still a substantial improvement, but it changes the cost arithmetic teams use to justify hardware migration decisions.

The metric that matters most for training is not peak flops but sustained memory bandwidth. Large language model training is memory-bandwidth-bound for the majority of operations. Attention layers, layer normalization, activation functions, and optimizer state updates all hit the memory subsystem harder than the compute units. The A100 delivers 2.0 TB/s of HBM2e bandwidth. The H100 SXM delivers 3.35 TB/s of HBM3. That 1.67x bandwidth improvement is often a better predictor of real-world training speedup than the 3x difference in peak teraflops. Teams evaluating gpu for deep learning workloads should profile their models with tools like PyTorch Profiler or Nsight Systems to determine whether their bottleneck is compute-bound or memory-bound before choosing hardware. Most transformer-based models at batch sizes teams actually use in production are memory-bound, which means the bandwidth number on the spec sheet is the one that predicts your experience.

Precision format support is another area where published benchmarks create confusion. An H100 can run FP8 inference at nearly double its FP16 throughput, but only if your model has been quantized correctly and your serving framework supports FP8 kernels end to end. BF16 has become the default training precision because it offers the same dynamic range as FP32 with half the memory footprint, and both the A100 and H100 support it natively. FP16 training is faster on paper but requires loss scaling to avoid underflow, which adds complexity and failure modes. When reading any gpu benchmark comparison that reports FP8 or FP16 numbers, ask whether the benchmark used the same precision format you plan to use. A benchmark showing 2x throughput improvement at FP8 is irrelevant if your production pipeline runs at BF16. Teams that are serious about gpu for deep learning performance need to benchmark at the precision they will actually deploy, not the precision that produces the best chart.

Inference benchmarks introduce a different set of problems. The metrics that matter for serving are time to first token, tokens per second per GPU, and throughput at a given latency percentile, typically p99. Published benchmarks almost never report p99 latency because it makes every GPU look worse. They report median latency or total throughput without latency constraints. In production serving, your GPUs will be handling concurrent requests at varying sequence lengths, and the variance in response time matters as much as the average. An a100 vs h100 comparison for inference should be run at your target request rate, with your model, at your quantization level, behind your serving framework. Anything else is a number that applies to someone else's workload. Teams running high-volume inference endpoints should evaluate hardware the same way they evaluate interconnect decisions, by measuring what actually changes under their specific operating conditions.

The only gpu benchmark comparison that means anything for your organization is one you run yourself. Profile your training job on both hardware generations for at least 1,000 steps with realistic batch sizes. Measure wall-clock time, GPU utilization, memory bandwidth utilization, and communication overhead if you are running multi-node. For inference, run your model at your target queries per second and measure time to first token and p99 latency across a sustained load test. These numbers will not match any published benchmark, and that is exactly the point. The published numbers exist to sell hardware. Your numbers exist to make a procurement decision. Teams that skip this step and rely on spec sheets routinely overspend on hardware that does not deliver the expected improvement, or worse, underspend on hardware that would have paid for itself in reduced total training cost. Run your own workloads, measure your own metrics, and let those numbers drive the decision.