Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Clear out junk files and repair common Windows errors3Scan for outdated or missing drivers - takes under a minuteiTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Benchmark AI inference hardware as a complete serving system, not as an accelerator in isolation. Use the same model, precision, prompt and output distributions, software stack, and quality target on every system; measure time to first token (TTFT), end-to-end latency, inter-token speed, throughput, failures, and the cost of reaching a defined service level. A peak tokens-per-second figure without its workload and latency is not a useful buying decision.
The practical method is to define the application and service target first, warm each system, sweep concurrency or offered request rate, record a load curve, and compare systems at the same operating point. Cost then becomes meaningful: cloud rental, amortized ownership, power, and other expenses are divided by successful useful output at that measured operating point.
What a fair AI inference benchmark must answer
A benchmark should answer three separate questions: How quickly does one request become visible and complete? How much work can the system handle as demand rises? What does that service level cost? These questions are related but not interchangeable.
- Responsiveness: TTFT tells an interactive user how long the system waits before the first streamed token. End-to-end latency tells when the complete answer arrives. ITL or TPOT tells how smoothly tokens arrive after streaming begins.
- Capacity: Aggregate output-token throughput and successful requests per second show how much total work the system completes. Per-user speed can decline while aggregate throughput rises as concurrency increases.
- Economics: Cost per request or output token is valid only when its accounting boundary, utilization, quality condition, and latency target are stated.
NVIDIA’s metrics documentation defines TTFT as the interval from query submission to the first received token and end-to-end latency as submission to receipt of the full response. Use those definitions or document any different interpretation. NVIDIA NIM metrics
#1 Best Overall
- Various Monitoring Parameters: The power meter plug can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD Display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
Do not call a vendor’s configured run a hardware-only result. Runtime, driver, kernel selection, batching, quantization, parallelism, and power limits are part of the tested system.
Choose the decision and service target before testing
Write down what the deployment must do before selecting a test. An interactive assistant, an offline summarization queue, and an embedding service need different operating points.
Interactive streaming
Set a percentile ceiling for TTFT, a maximum ITL or TPOT (or a minimum per-user token rate), and a completion-latency objective. A median alone can hide users experiencing queueing or long prompts, so include a high percentile such as P95 when the service-level objective requires it.
Free tools Windows power users keep installed
One-click scans. No signup required.
Batch or offline generation
Define a completion window, required output-token rate, and acceptable failure or retry policy. TTFT may be less important than total completed tokens per hour, but report latency so a future interactive use is not inferred from a batch result.
Capacity planning
Define the expected request-rate or concurrency range and the maximum latency that may not be breached. MLPerf Endpoints presents system throughput, interactivity, P95 TTFT, and concurrency as related dimensions of an operating point. MLPerf Endpoints
Record the target in the test plan, for example: maintain P95 TTFT below the chosen ceiling and P95 TPOT below the chosen limit while completing the required requests per second. The exact values are application requirements; they are not universal hardware specifications.
Measure the right latency and throughput metrics
Use a consistent tool and version, and publish formulas when a metric could be interpreted more than one way. NVIDIA states that tool implementations vary, so compare results only when definitions align. NVIDIA NIM Benchmarking Documentation
| Metric | What it measures | How to use it | Important qualification |
|---|---|---|---|
| TTFT | Submission until the first output token is received. | Primary interactive responsiveness indicator; include percentiles. | Includes prompt processing and waiting before visible streaming. Requests with no output token need a stated treatment. |
| End-to-end latency | Submission until the complete response is received. | Shows total user wait, including generation and queueing. | NVIDIA’s documented definition includes queueing, batching, and network latency. |
| ITL / TPOT | Average interval between generated tokens, or time per output token. | Describes streaming smoothness and per-user generation speed. | Definitions differ: verify whether TTFT is excluded and how requests are weighted. TensorRT-LLM serving benchmark notes |
| System output TPS | Aggregate generated output tokens per second across concurrent requests. | Capacity metric for sizing and cost calculations. | It can increase with concurrency until compute saturates, then flatten or fall. |
| Requests per second | Successfully completed requests divided by the measured interval. | Useful for API capacity and queue sizing. | Lengths vary; always report input and output token distributions alongside it. |
| Power or energy | Electrical consumption over a stated boundary and interval. | Converts performance into energy or operating cost. | Specify accelerator-only versus whole-system readings and whether measurement is at the wall. |
For streaming systems, report TTFT, end-to-end latency, and ITL or TPOT together. A system can have excellent TTFT but slow token pacing, or high aggregate TPS while individual users wait longer.
Rank #2
- Multi-function power monitor: Our electric usage monitor can monitor the power (W), electricity(kWh), voltage(V), frequency(Hz), current(A), power factor(PF), unit price($/kWh), total cost($) of your appliances. By switching 8 display modes, you can easily know the various parameters while the appliance is working. The “electricity” mode can calculate and display how much power your appliance uses. And the “total cost” mode will show how much electricity bill it cost in cumulative time
- Overload protection: When the loading power of the appliance is over the default overload threshold 1800W, the whole display with backlight and the word “OVERLOAD” will keep flashing to warn the users. Please turn off the appliance for safety concern
- Premium Material: The whole body of our energy meter is made of high-quality ABS material. It makes our home electricity usage monitor more long lasting, fireproof and anti-drop. The standard US socket and plug is suitable for all US standard appliances
- Backlight Display: With white backlight and black words, the LCD display of our home power monitor can display the data clearer and help you to read the data easier no matter day or night. The backlight will only lights up when the device is connected to AC power. If no button is pressed, the backlight turns off automatically after 10 minutes. You can also press the "UP" button to turn off the backlight manually, and press any button to turn on backlight again
- Easy to reset: No reset tool needed, our appliance power usage meter is easy to reset. You can press the “M” button for 5 seconds directly to reset the device. After reset, all cumulative data (electricity quantity, cost) will be cleared, and all settings will be restored to factory settings
Fix the workload so systems are comparable
The workload is part of the benchmark result. Capture every variable that changes the amount or shape of computation.
Model and quality
- Model name, exact revision or checkpoint, tokenizer, and context limit.
- Precision or quantization format, calibration method when applicable, and any quality threshold.
- Decoding settings such as temperature, top-p, maximum output tokens, stop conditions, and whether output length is fixed or ends at end-of-sequence.
Prompt and response distributions
Record input and output token-length distributions, not merely an average. Include the dataset or prompt source and the proportion of short, medium, and long contexts. Comparing one accelerator on short prompts with another on long context measures the workload difference as much as the hardware difference.
Synthetic prompts can isolate a system behavior, but disclose that they are synthetic and make them resemble production traffic before using the result for procurement. Keep random seeds or prompt files so another run can reproduce the same requests.
Traffic pattern
State whether requests arrive at a fixed rate, in bursts, or as an open-loop stream; the concurrency range; request timeout and retry rules; and whether a client waits for completion before sending another request. Note streaming versus non-streaming API behavior and network placement.
Record the complete system configuration
Two machines with the same accelerator model can perform differently because of the rest of the serving stack. Record:
- Accelerator model, count, memory capacity, interconnect, clock or power limits, and host CPU and RAM.
- Driver, operating-system, CUDA or equivalent runtime, inference engine, model server, and benchmark-client versions.
- Tensor and pipeline parallelism, replica count, batch-size limits, continuous-batching or scheduling policy, queue limits, and admission control.
- Kernel, attention, speculative-decoding, graph-capture, compilation, and other optimization settings.
- Precision, quantization, and model conversion steps, including whether a vendor-provided engine was used.
- Network topology and link speed when the client and server are separate, plus storage or model-loading behavior.
NVIDIA’s TensorRT-LLM workflow starts from a prepared dataset and distinguishes throughput and low-latency benchmark modes. Treat those modes as different experiments rather than mixing their results. TensorRT-LLM performance benchmarking
Run the benchmark: a repeatable procedure
- Prepare an immutable test record. Save model files, tokenizer, prompt set, decoding configuration, software versions, hardware inventory, and the exact service target. Record the date and region for any cloud run.
- Validate correctness first. Confirm that every system produces acceptable output quality, honors stop conditions, and uses the intended precision. A faster system that returns truncated or lower-quality answers is not an equivalent result.
- Warm up the service. Send enough requests to load the model, compile kernels, allocate memory, and reach normal cache and scheduler behavior. Exclude warm-up from the measured interval and state how many requests or how much time it used.
- Measure a low-load point. Start at one request or low concurrency to expose single-user TTFT, TPOT, and end-to-end latency without queueing.
- Sweep load in controlled steps. Increase concurrency or offered request rate step by step. Hold each level for a defined steady-state window, discard failed or timed-out requests from successful-throughput calculations, and count them separately.
- Collect the same fields at every level. Record successful requests, output tokens, input and output lengths, TTFT, ITL or TPOT, end-to-end latency, queue time when available, and errors or timeouts. Report median and high percentiles appropriate to the service target.
- Stop at the decision boundary. Continue until the latency target is breached, errors appear, or additional load no longer increases completed throughput. State whether throughput is offered, completed, or measured only during a steady-state window.
- Repeat for variance. Run independent repetitions or longer windows when results fluctuate. Report the spread and environmental conditions instead of selecting a favorable run.
A load curve is more informative than one maximum number. It shows where aggregate TPS rises, where per-user pacing degrades, and where the service target is crossed.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallCompare systems at matched operating points
Never compare a maximum-throughput run from one system with a low-latency run from another. Use one of these methods:
Rank #3
- Various Monitoring Parameters: The power energy meter can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD Display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload Protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
- Shared service target: Select the highest completed throughput each system achieves while meeting the same TTFT, TPOT, and quality limits.
- Full curves: Plot throughput against latency for every load level and identify the target region. This is useful when no single target represents all customers.
- Standardized scenario: Use the applicable MLPerf workload, scenario, quality target, and rules for the specific round, then report the submitted configuration and availability category.
MLPerf Inference is designed for architecture-neutral, representative, reproducible comparisons with published rules and reference software. Its results answer a standardized question; they do not replace an application benchmark using your traffic and serving stack. MLCommons MLPerf Inference
Benchmark definitions change. MLCommons announced v6.1 on September 16, 2026, including new tests for emerging inference patterns. The announcement reports a 2.99× improvement for the best per-accelerator server-scenario result in its VLM benchmark over v6.0; that is a round-specific result, not a universal hardware gain. Verify the current rules, result artifact, workload, and system availability before quoting a leaderboard. MLPerf Inference v6.1 announcement
Calculate inference cost without hiding assumptions
Choose the accounting boundary first. A cloud comparison may use rental or service price; an owned-system comparison may amortize purchase cost; a total-cost model may add power, cooling, space, support, and administration. Do not mix boundaries.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Cloud rental
Use the actual instance or service price, region, billing unit, and pricing date. Include idle time, storage, data transfer, and minimum billing increments when they apply. State utilization: a server billed 24 hours but serving traffic for only a fraction of that time has a different effective cost per token.
Owned hardware
State purchase price, amortization period, expected utilization, residual-value assumption if any, and included operating costs. A one-time quote is not a universal hardware price. If price is unavailable, report cost components separately rather than inventing a figure.
Cost per request or output token
For a measured period, calculate:
cost per output token = total cost assigned to the period ÷ successfully generated output tokens in that period
Use the same operating point, quality condition, and successful-output definition for every system. For request cost, divide by successfully completed requests, while still reporting token lengths because requests with very different output sizes are not equivalent.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →Power and energy boundary
Specify whether readings cover the accelerator, host, or entire system and whether they are measured at the wall. MLPerf Datacenter describes average AC power for the entire system measured at the wall during the performance measurement. MLPerf Datacenter results and power methodology
Rank #4
- ✓ Backlit LCD Display: The watt meter features an upgraded backlit LCD display with a 160° wide viewing angle, providing clear and distinct data readings from any angle, day or night. (Note: To conserve energy, the backlight will automatically turn off after 15 seconds of continuous use.)
- ✓ 9 Types of Data Display: This energy meter plug accurately monitors a wide range of electrical parameters, including power (W), apparent power (VA), energy consumption (kWh), usage time (hours), power frequency (Hz), power factor (PF), voltage (V), current (A), and electricity cost. The user-friendly interface allows you to easily read and navigate through 9 types of data with a single button.
- ✓ Save Money and Energy: By toggling through the 9 display modes, you can gain insights into various operational parameters of your devices. The home energy monitor also calculates and displays the electricity consumption and cost over time, helping you save on your electricity bills by monitoring and managing energy usage.
- ✓ Data Storage: The plug-in electricity meter stores all non-real-time data, ensuring no information is lost in the event of a sudden power outage. If data exceeds the screen’s maximum display range, the screen will flash as a reminder. Pressing and holding the reset button for 3 seconds will reset the cumulative data to zero.
- ✓ Safe to Use: Rated voltage 125VAC, 60Hz, 15A maximum 1875W resistor and tungsten wire, 1/2HP. Clean and tidy interface, very simple to use.
A plug-in AC power meter can be suitable for a compatible workstation, but it is not automatically appropriate for a multi-GPU server or rack circuit. Check electrical ratings, measurement accuracy, connector type, and facility requirements before using one.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Use a comparison matrix that preserves context
For each system, keep one row per matched operating point rather than one row per product. Include:
| Comparison field | What to enter |
|---|---|
| Workload | Model revision, tokenizer, precision, prompt and output distributions, decoding, dataset, and quality test. |
| System | Accelerator model and count, host, memory, interconnect, power limit, and availability category. |
| Software | Driver, runtime, inference engine, server, client, benchmark tool and version, parallelism, batching, and scheduler settings. |
| Load | Concurrency or offered request rate, warm-up method, measurement window, and timeout policy. |
| Latency | Median and P95 or other required percentiles for TTFT, ITL/TPOT, and end-to-end latency. |
| Throughput | Successful requests per second, aggregate output TPS, input and output tokens, and failed or queued requests. |
| Cost | Accounting boundary, utilization, period, cost per request or output token, and power measurement boundary. |
This matrix prevents a favorable number from being detached from the conditions that produced it.
Why your result differs from a vendor benchmark
Different token lengths or decoding
Longer contexts increase prompt processing and memory pressure; longer outputs increase generation time. Different stop behavior, temperature, or speculative decoding also changes work. Re-run with the same prompt and output distributions.
Different metric definitions
Tools may include or exclude TTFT in ITL, use different benchmark-duration boundaries, or weight requests differently. NVIDIA specifically notes differences between GenAI-Perf and LLMPerf. Use one tool consistently or publish formulas and versions with the result. NVIDIA LLM benchmarking fundamentals
Warm-up and steady state
A cold start, kernel compilation, model loading, or changing cache state can dominate a short run. Separate warm-up and use a window long enough to represent normal service. Conversely, an excessively long run can include thermal or power behavior absent from a vendor’s interval; disclose the difference.
Batching, queueing, and concurrency
Throughput generally rises as concurrent work fills the accelerator, while per-user latency worsens. Different batch limits or schedulers can therefore produce opposite rankings at low and high load. Compare the same target and show the curve.
Power and system boundaries
An accelerator telemetry reading is not the same as whole-system wall power. Host CPU, memory, fans, networking, and power-delivery losses matter for operating cost. Match boundaries before comparing energy per token.
Best Value
- Various Monitoring Parameters: The power meter plug can monitor the power (W), energy (kWh), volts, amps, hertz, power factor, cost, minimum and maximum power (W), cumulative days and time of your appliances. By switching 7 display modes, you can easily know the various parameters while the appliance is working. The home energy monitor can also calculate and display how much power your appliance uses and how much electricity bill it cost in cumulative time
- Upgraded LCD display: With large screen size 2.36 inch x 1.85 inch, clearer monitor backlit, our electrical usage monitor can display the data clearer and more visible no matter day or night. 180°full wide viewing angles is great for reading and recording the data in any angles. No need to stand on the front of the display and bend over to read the numbers
- Adjustable Backlight Time: Our upgraded watt meter has 5 options of backlight time. The default backlight time duration is 10 minutes(bL-0). If you want to change the backlight time, you can press and hold "UP" and "DOWN" button at the same time to enter backlight time setting, then press "UP" and "DOWN" to select the backlight time (bL-0 =10 minutes, bL-1=1 hour, bL-2=4 hours, bL-3=8 hours, bL-4=always on), finally press the "COST" to save the backlight time settings
- Overload protection: When the power of the appliance exceeds the overload power, the LCD will display “OVERLOAD” to warn the user. All the buttons will quit working and can only be workable when you lower or remove the load power. The default overload power is 3680W and is adjustable from 0 to 3680W. In general, you need to set the overload power to 1800W before using. Just press the "function" button for more than 3 seconds to enter the setting
- Data Memory Function: The wattage meter will record your power consumption data when you remove it from socket, or remove appliances from the electricity monitor. You can directly see the last data when you use it next time. This function can also automatically save the data when there is a sudden power failure
Quality or availability mismatch
A benchmark may use a different quality target, model variant, or system category. Check the exact MLPerf round, scenario, submission rules, and availability status before treating a published figure as applicable to your deployment.
How to report the result so another team can reproduce it
Publish a compact test record with these fields:
- Test date, location or cloud region, and ambient or facility conditions that materially affected operation.
- Model revision, tokenizer, precision, quantization, decoding, prompt source, and token-length distributions.
- Hardware inventory, topology, power limits, software versions, inference engine, server, client, and benchmark tool.
- Warm-up procedure, load steps, steady-state duration, timeout and retry rules, and handling of queued or failed requests.
- At each load: successful request rate, output TPS, TTFT, ITL/TPOT, end-to-end latency percentiles, error count, and token counts.
- Cost boundary, pricing date or amortization assumptions, utilization, power instrumentation, and the exact operating point used for cost per token.
A CSV or database export should retain one record per request when privacy permits, plus an aggregated table per load level. That makes percentile calculations, outlier review, and later re-analysis possible.
Frequently asked questions
How many concurrent users can a GPU serve?
There is no hardware-only number. The answer is the highest concurrency that meets your model, token-length distribution, quality condition, and latency target. Run the load sweep and report the target-conforming operating point.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
What is the difference between TTFT and tokens per second?
TTFT is the wait until the first token. Tokens per second can mean aggregate system output TPS or a per-user generation rate derived from ITL or TPOT. Report both because a high aggregate rate can coexist with slow individual responses.
Should I use MLPerf or an application benchmark?
Use both when possible. MLPerf provides a standardized, reproducible comparison under defined rules; an application benchmark reveals how your model, traffic shape, service target, and software stack behave.
Can I compare cost per token from two cloud providers?
Yes, if both figures use the same model and workload, successful-output definition, latency or quality target, utilization, pricing date and region, and accounting boundary. Otherwise the numbers describe different services.
Frequently Asked Questions
How often should an inference benchmark be rerun?
Rerun after changing the model, precision, serving runtime, driver, scheduler, hardware configuration, or pricing assumptions. Also rerun when a standardized benchmark round changes, because workloads and rules are versioned.
Recommended Free Tools
What should I do when benchmark runs are unstable?
Check thermal and power limits, background processes, request-rate control, queue depth, warm-up, and measurement-window length. Repeat independent runs, report variance, and investigate failed or timed-out requests instead of discarding them silently.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
