How to Benchmark AI Workloads Across GPUs and Custom Accelerators

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

The only fair way to compare GPUs, TPUs and other custom accelerators is to benchmark the complete system on the workload you actually plan to run. Define the model, data shapes, precision, quality target and service or training goal first; then measure latency, throughput or time-to-quality under a documented hardware and software configuration. A peak number detached from those conditions is not a portable ranking.

What an AI accelerator benchmark should answer

A benchmark is useful only when it answers a specific capacity, architecture or purchasing question. “Which chip is fastest?” is too vague. A defensible test answers a question such as:

#	Preview	Product	Price
1		NVD RTX PRO 6000 Blackwell Professional Workstation Edition Graphics Card for AI, Design,...	$19,999.99	Buy on Amazon
2		ASRock Intel Arc Pro B70 Creator 32GB Workstation Graphics Card, Xe2-HPG, 32GB GDDR6, PCIe 5.0, 4X...	$1,299.99	Buy on Amazon
3		ASRock Radeon AI PRO R9700 Creator 32GB Professional Graphics Card, 2920 MHz Boost Clock, GDDR6, AMD...	$1,699.99	Buy on Amazon
4		ASRock Intel Arc Pro B60 Creator 24GB Graphics Card, Workstation GPU, Xe2-HPG, 2400MHz, 24GB GDDR6...	$649.99	Buy on Amazon
5		PNY NVIDIA RTX A6000	$5,945.17	Buy on Amazon

• Which system trains this model to a specified validation quality in the shortest elapsed time?
• Which configuration serves requests within a 100-millisecond p95 latency target at a stated concurrency?
• What is the cost per million generated tokens at a defined output quality?
• How does throughput change when the deployment grows from one accelerator to eight or more?

Google Cloud’s cross-accelerator benchmarking guidance uses NVIDIA GPUs, AMD accelerators, Google TPUs and AWS accelerators as examples, with large-language-model workloads illustrating why architecture and workload must be considered together. MLPerf’s inference definitions make the same principle concrete by tying each test to a dataset, scenario and quality target. A result without those particulars is an answer to a different question.

1. Define the production workload before choosing a test

Training or inference

Training and inference stress different parts of a system. Training usually involves repeated forward and backward passes, optimizer state, checkpointing and collective communication. Inference may be dominated by memory capacity, token generation, request scheduling, host preprocessing or tail latency. State which phase you are testing; do not compare a training step rate with an inference requests-per-second figure.

#1 Best Overall

NVD RTX PRO 6000 Blackwell Professional Workstation Edition Graphics Card for AI, Design, Simulation, Engineering - 96GB DDR7 ECC Memory - 4th Gen RT/5th Gen Tensor Core GPU - OEM Packaging

• ATTENTION BUYERS: EXPORTING this item outside the U.S. requires strict compliance with the Bureau of Industry and Security (BIS) U.S. Export Administration Regulations (EAR). Non-compliance carries fines up to $1 MILLION PER VIOLATION AND UP TO 20 YEARS IMPRISONMENT for both the buyer and seller. To uphold our regulatory obligations, we strictly follow BIS "Know Your Customer" guidance by screening each order and immediately aborting transactions showing unresolved regulatory red flags. ***
• [Double-Flow-Through Design] The RTX PRO 6000 Blackwell features a double-flow-through cooling design, optimizing efficiency and airflow to sustain peak performance under 600W power loads. | [5th Gen Tensor Cores] Deliver up to 3X the performance of the previous generation and support for FP4 precision for faster AI model processing times with reduced memory usage, enabling local fine-tuning of LLMs and generative AI | [4th Gen Ray Tracing Cores] Double the ray-triangle intersection rate of the previous generation to create photoreal, physically accurate scenes and immersive 3D designs with RTX Mega Geometry, which enables up to 100X more ray-traced triangles.
• [PCIe Gen 5] Support for PCIe Gen 5 provides double the bandwidth of PCIe Gen 4, improving data-transfer speeds from CPU memory and unlocking faster performance for data-intensive tasks like AI, data science, and 3D modeling. | [GDDR7 Memory] With 96 GB of GPU memory and 1.8 TB ps bandwidth, it can tackle massive 3D and AI projects, fine-tune AI models locally, explore large-scale VR environments, and drive larger multi-app workflows.
• [DisplayPort 2.1] Achieve unparalleled visual clarity and performance, driving high resolution displays at up to 8K at 240 Hz and 16K at 60 Hz. Increased bandwidth enables seamless multi-monitor setups while HDR and higher color depth support ensures superior color accuracy for precision work, such as video editing, 3D design, and live broadcasting.
• [NVIDIA Blackwell Streaming Multiprocessor] The new SM features increased processing throughput, and new neural shaders that integrate neural networks inside of programmable shaders | DLSS 4: Multi Frame Generation ensures ultra-smooth frame pacing for lifelike simulations. | [Universal MIG] Divide a single RTX PRO 6000 Blackwell into multiple isolated instances, each with dedicated resources, allowing for concurrent execution of multiple workloads, optimized GPU utilization, and secure isolation of different applications or users. [WARRANTY] 3 YR Manufacturer's Warranty. Bulk OEM Packaging. Retail Packaging is NOT included.

$19,999.99

Buy on Amazon

Model and version

Record the exact model family, parameter count, checkpoint or commit, tokenizer and framework implementation. A change in attention kernel, quantization method or checkpoint can alter both speed and output quality. If you use a representative model rather than the final production model, explain which architectural and dimensional properties are intended to match.

Input, output and request pattern

For language models, specify prompt length, generated-token limit and the distribution of sequence lengths. For vision, speech or multimodal systems, record image dimensions, audio duration, sample rate and modality mix. State batch size, concurrent requests and arrival pattern. A benchmark using fixed, short prompts can make a device look faster than it will be for long-context traffic.

Precision and quality target

List data types for weights, activations, accumulation and communication, such as FP32, BF16, FP16, INT8 or a lower-bit format. Reduced precision and accelerator-specific kernels can change numerical behavior. Establish the acceptable quality target before running the speed test, then verify accuracy, perplexity, pass rate or another task metric on the same evaluation set. MLPerf Inference explicitly defines tests around datasets and quality targets for this reason.

Decision and acceptance criteria

Write the decision rule in advance. For example, “select the lowest-cost system that sustains 2,000 requests per second at p95 latency below 150 ms and meets the agreed quality score,” or “select the system with the lowest time to validation loss X.” Without an acceptance rule, teams tend to select whichever number is easiest to publicize.

2. Declare a fair measurement boundary

Decide what the clock includes and apply the same rule to every accelerator. A kernel-only timer can be appropriate for a microbenchmark, but it does not represent an application if tokenization, host-to-device copies, synchronization or postprocessing are on the critical path.

• Preprocessing: tokenization, image decoding, feature extraction and input validation.
• Transfers: host-to-device and device-to-host copies, staging through system memory and network movement between workers.
• Execution: accelerator kernels, graph compilation and runtime scheduling.
• Synchronization: barriers, all-reduce operations, collective communication and queue waits.
• Postprocessing: detokenization, decoding, ranking and response serialization.

Publish separate numbers when a boundary is ambiguous. For example, report accelerator execution latency and end-to-end service latency rather than silently mixing them. Google Cloud’s methodology calls out host-to-device and device-to-host transfer rates as relevant benchmark dimensions; those transfers can decide the result when the model or request is small.

Rank #2

ASRock Intel Arc Pro B70 Creator 32GB Workstation Graphics Card, Xe2-HPG, 32GB GDDR6, PCIe 5.0, 4X DP 2.1, Blower Fan, Vapor Chamber, Honeywell PTM7950

• System Compatibility Note: This 2-slot card measures 271 x 112 x 39 mm and requires a single 12V-2x6-pin power connector. Please verify chassis and PSU compatibility before purchase.
• Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
• Professional Intel Arc Pro B70 GPU: Built on the Intel Xe2-HPG architecture, it features 32 Xe cores and 256 XMX engines, designed to accelerate AI, rendering, and complex visualization workloads.
• Massive 32GB GDDR6 VRAM: Equipped with 32GB of high-speed GDDR6 memory on a 256-bit bus, running at 19 Gbps, which allows for handling large AI models and complex datasets locally.
• High-Performance Engine Clock: Delivers an engine clock of 2540 MHz, providing the compute power needed for demanding professional applications and AI inference.

$1,299.99

Buy on Amazon

3. Record the entire system, not just the accelerator name

The delivered result belongs to a system. Two identical accelerators can produce different results because of host CPUs, memory, interconnect topology, driver versions or software libraries.

Include firmware and configuration details that affect determinism, such as power limits, fan mode, persistence mode, compiler flags and environment variables. For a cloud run, record region, machine shape, attached storage and whether the instance is dedicated or shared.

4. Choose metrics that match the workload

Inference: latency and throughput at a stated operating point

Report median and tail latency, normally p95 and p99, together with throughput. State concurrency or request arrival rate, prompt and output lengths, batching policy and whether the measurement is steady-state. A single unconstrained requests-per-second maximum can hide unacceptable tail latency or an unrealistic batch size.

For generative models, separate time to first token from inter-token latency and total response time when those measures matter to users. Define whether queueing time is included. If a service has several traffic classes, test each representative mix instead of publishing one averaged request.

Training: time to quality, not only steps per second

Report samples or tokens per second, completed-step time, device utilization and elapsed time to a defined loss or accuracy target. Raw throughput can favor an optimization that requires more steps or produces a different convergence path. The MLPerf Training Benchmark paper identifies stochastic variation, hardware and software diversity, and the difference between throughput optimization and time-to-solution as central fairness challenges.

For multi-device training, report scaling efficiency in addition to absolute throughput. Define it explicitly, for example as measured throughput divided by ideal linear throughput from the one-device baseline, and state whether the baseline uses the same batch and optimization settings.

Rank #3

ASRock Radeon AI PRO R9700 Creator 32GB Professional Graphics Card, 2920 MHz Boost Clock, GDDR6, AMD RDNA 4, AI-Accelerators, DisplayPort 2.1a, PCIe 5.0, Blower Cooler

• Professional AI & Creator Workstation: AMD Radeon AI PRO R9700 GPU with 32GB GDDR6 is engineered for AI development, professional content creation, and compute-intensive workloads.
• Massive 32GB Memory Capacity: 32GB of GDDR6 memory on a 256-bit bus provides ample bandwidth for large AI models, 8K video editing, and complex 3D rendering.
• Advanced RDNA 4 with AI Accelerators: 64 Compute Units with 3rd Gen Ray Tracing and dedicated 2nd Gen AI Accelerators for groundbreaking AI performance and visual computing.
• Professional Blower Cooling: Efficient single blower design exhausts heat directly out of the chassis, ideal for multi-GPU workstation and server configurations.
• Enterprise-Grade Thermal Solution: Vapor chamber heatsink with industrial Honeywell PTM7950 thermal interface material ensures reliable cooling under sustained professional loads.

$1,699.99

Buy on Amazon

Cost and energy

Cost per request, token, sample or completed training run is useful only at the same quality and service target. State the price source, region, billing term, accelerator and host charges, networking, storage, orchestration and idle capacity. If you report energy, identify the meter boundary and interval. A device-only wattage figure cannot be compared with a whole-node measurement.

5. A repeatable benchmark procedure

1. Write the workload specification. Name the model and version, dataset, input and output shapes, precision, batch or concurrency, quality metric and acceptance target.
2. Freeze the software environment. Pin the operating system, driver, runtime, compiler, framework, libraries and benchmark commit. Save container or environment definitions.
3. Verify correctness first. Run a small case and compare outputs, loss or task metrics with a trusted reference. Investigate numerical drift before collecting performance numbers.
4. Measure initialization separately. Record compilation, graph capture, model loading and memory allocation time. Exclude them from steady-state figures only when production amortizes them; otherwise report both.
5. Warm up the system. Use a stated number of requests or steps until compilation and caches settle. Keep the warm-up policy identical across systems.
6. Run the production-like load. For inference, sweep the relevant concurrency or arrival-rate range. For training, run enough steps or complete runs to reach the quality target rather than stopping at the first attractive throughput sample.
7. Repeat and retain all runs. Report median, spread and outliers. Do not publish only the fastest repetition.
8. Collect utilization and bottleneck counters. Capture accelerator utilization, memory use and bandwidth, host utilization, transfer rates, communication time and queue depth where the platform exposes them.
9. Repeat at additional device counts. Measure one, two, four and the intended production scale when feasible. Keep model, quality and software settings comparable and document any necessary changes.
10. Publish the configuration with the result. Include the test boundary, formulas, repetitions, exclusions and raw or summarized measurements so another team can reproduce the comparison.

6. Test scaling instead of extrapolating it

A one-accelerator result does not predict an eight-accelerator deployment. As device count rises, communication, synchronization, memory pressure and host orchestration can consume the gains from parallel execution. Report both absolute throughput and scaling efficiency at each tested size.

Use traces or counters to determine whether the limit is compute, memory bandwidth, interconnect, input delivery or software utilization. A low device-utilization percentage can mean the model is memory-bound, but it can also indicate a slow data loader or excessive synchronization. Test topology changes, placement and communication settings when they are part of the production configuration.

For training, preserve a valid optimization and convergence setup while scaling. Increasing global batch size may improve throughput but alter the number of steps, learning-rate schedule or final quality. Compare elapsed time to the same target, not merely tokens per second. For inference, scaling can be vertical (larger batches on one host) or horizontal (more replicas); measure queueing and tail latency for the strategy you will deploy.

7. Preserve quality and reproducibility across different stacks

Different accelerators rarely use an identical binary. A GPU may use one kernel library, a TPU another compiler path, and a custom accelerator a specialized graph partitioner. That is acceptable, but every optimization must be documented and validated against the quality target.

• Keep the evaluation dataset and scoring code fixed.
• Record random seeds, sampling settings and deterministic-mode choices.
• Report quantization calibration data and calibration procedure.
• Identify fused operators, sparsity, speculative decoding, recomputation or other non-default optimizations.
• State whether weights, activations or communication use different precision from the reference.
• Repeat training runs when stochastic variation is material, and show the variation rather than treating one run as definitive.

Reproducibility also requires recording data versions and preprocessing. “Same model” is not enough if tokenization, image resizing or filtering differs. Publish a small correctness report alongside performance so readers can see that a faster configuration did not silently change the task.

Rank #4

ASRock Intel Arc Pro B60 Creator 24GB Graphics Card, Workstation GPU, Xe2-HPG, 2400MHz, 24GB GDDR6 192-bit, PCIe 5.0, 4X DP 2.1, Blower

• System Compatibility Note: 2-slot card, 271x112x39mm, single 8-pin power, 200W TDP. Verify chassis clearance and PSU capacity before purchase.
• Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
• 24GB GDDR6 on 192-Bit Bus: Massive 24GB memory with 456 GB/s bandwidth – ideal for LLMs, AI inference, 3D rendering, and generative design.
• Intel Xe2-HPG Architecture: Built on Intel's next-gen architecture with 20 Xe cores and 160 XMX engines for AI acceleration (197 INT8 TOPS).
• PCIe 5.0 Support: PCI Express 5.0 x16 interface for maximum bandwidth with the latest workstation platforms.

$649.99

Buy on Amazon

8. How to interpret published accelerator figures

Vendor benchmarks are useful evidence about the tested configuration, not universal rankings. Keep the model, benchmark version, system size, precision, quality target and price assumptions attached to every number.

When reading any published chart, ask whether it measures kernel time or end-to-end time, whether the quality target is equivalent, and whether the compared systems use the same number of devices. A chart that omits those details can still be a useful lead for a reproduction, but it should not be used as a purchasing conclusion by itself.

9. Present results in a comparison table readers can audit

Keep unlike experiments in separate rows or separate tables. A practical report contains at least these columns:

Include a note for any cell that is not established. “Not stated” is more honest than filling a gap with an estimate. If a vendor used a different benchmark version or system size, keep that qualification in the same row as the number.

10. Diagnose common benchmark failures

The accelerator is busy but throughput is low

Check whether the workload is memory-bandwidth-bound, whether batch or sequence lengths fit the device efficiently, and whether precision-specific kernels are available. Compare achieved memory bandwidth with the platform capability and test a representative range of shapes rather than one ideal shape.

Utilization is low and latency is erratic

Inspect host preprocessing, data-loader workers, queue depth, synchronization and transfer times. Small requests can spend more time crossing the host-device boundary than executing kernels. Measure end-to-end traces before changing the model.

Best Value

PNY NVIDIA RTX A6000

• NVIDIA Ampere Architecture-based CUDA Cores - Double-speed processing for single-precision floating point (FP32) operations and improved power efficiency provide significant performance improvements for graphics and simulation workflows, such as complex 3D computer-aided design (CAD) and computer-aided engineering (CAE), on the desktop.
• Second-Generation RT Cores - With up to 2X the throughput over the previous generation and the ability to concurrently run ray tracing with either shading or denoising capabilities, second-generation RT Cores deliver massive speedups for workloads like photorealistic rendering of movie content, architectural design evaluations, and virtual prototyping of product designs. This technology also speeds up the rendering of ray-traced motion blur for faster results with greater visual accuracy.
• Third-Generation Tensor Cores - New Tensor Float 32 (TF32) precision provides up to 5X the training throughput over the previous generation to accelerate AI and data science model training without requiring any code changes. Hardware support for structural sparsity doubles the throughput for inferencing. Tensor Cores also bring AI to graphics with capabilities like DLSS, AI denoising, and enhanced editing for select applications.
• Third-Generation NVIDIA NVLink - Increased GPU-to-GPU interconnect bandwidth provides a single scalable memory to accelerate graphics and compute workloads and tackle larger datasets.
• 48 Gigabytes (GB) of GPU Memory - Ultra-fast GDDR6 memory, scalable up to 96 GB with NVLink, gives data scientists, engineers, and creative professionals the large memory necessary to work with massive datasets and workloads like data science and simulation.

$5,945.17

Buy on Amazon

Multi-device scaling collapses

Profile collective communication and network links, verify device placement and topology, and check whether the global batch or sequence partition is large enough to amortize synchronization. Compare scaling efficiency with the one-device baseline under the same quality settings.

The fast run fails the quality check

Revert to the reference precision or kernel path, then isolate quantization, calibration, sampling and compiler transformations one at a time. Keep the speed result only if the configuration meets the predeclared quality target.

Results vary between repetitions

Separate compilation and warm-up, check thermal or power throttling, control background activity and report the spread. Training has inherent stochastic variation; compare completed runs at the target quality when practical instead of selecting the luckiest run.

11. Local workstation or cloud: choose access based on the test

A local workstation can be convenient for repeatable single-device development, but it may not represent the host, network fabric or scale required in production. Cloud infrastructure can provide access to GPUs, TPUs or other accelerators and makes multi-device experiments easier, but region, capacity, storage and pricing affect cost and reproducibility.

Choose capacity from the model’s memory footprint, framework support, required device count, data location and geography—not from a headline compute rating. For either environment, retain the exact machine shape and software image. If the production system will use a managed serving layer, benchmark that layer or clearly label a lower-level test as a component measurement.

12. A decision framework for selecting the winner

1. Discard configurations that fail the quality target or cannot fit the workload in available memory.
2. Discard configurations that miss the required p95 or p99 latency, throughput or time-to-quality target.
3. Among the remaining systems, compare cost or energy at the same workload and quality, including the same resource boundary.
4. Check scaling behavior at the deployment size and leave capacity for realistic traffic or training variation.
5. Prefer the configuration with the clearest reproducibility and software support when performance and cost are close.

This process can select different winners for different workloads. A device that leads on large-batch training may lose on small, latency-sensitive inference; neither result contradicts the other. The meaningful conclusion is always conditional on the stated model, quality, system, measurement boundary and operating point.

Quick Recap

Bestseller No. 1

NVD RTX PRO 6000 Blackwell Professional Workstation Edition Graphics Card for AI, Design, Simulation, Engineering - 96GB DDR7 ECC Memory - 4th Gen RT/5th Gen Tensor Core GPU - OEM Packaging

$19,999.99

Bestseller No. 2

ASRock Intel Arc Pro B70 Creator 32GB Workstation Graphics Card, Xe2-HPG, 32GB GDDR6, PCIe 5.0, 4X DP 2.1, Blower Fan, Vapor Chamber, Honeywell PTM7950

$1,299.99

Bestseller No. 3

ASRock Radeon AI PRO R9700 Creator 32GB Professional Graphics Card, 2920 MHz Boost Clock, GDDR6, AMD RDNA 4, AI-Accelerators, DisplayPort 2.1a, PCIe 5.0, Blower Cooler

$1,699.99

Bestseller No. 4

ASRock Intel Arc Pro B60 Creator 24GB Graphics Card, Workstation GPU, Xe2-HPG, 2400MHz, 24GB GDDR6 192-bit, PCIe 5.0, 4X DP 2.1, Blower

$649.99

Bestseller No. 5

PNY NVIDIA RTX A6000

$5,945.17

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.