iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
“Tokens per second” can mean the combined output rate of a whole inference system, the generation cadence for one request, or the speed of processing an input prompt. The figure is useful only when you know which tokens are counted, what time interval is used, and how many requests were running. For an interactive user, read it alongside time to first token (TTFT), inter-token latency (ITL or TPOT), and end-to-end latency.
What does tokens per second measure?
At its simplest, tokens per second (TPS) is a token count divided by a time interval. But the label alone does not tell you whether the count covers generated output or prompt input, whether it is for one request or many, or where the timing interval begins and ends.
Those distinctions separate three commonly confused measures:
- Aggregate output TPS: output tokens produced across concurrent requests divided by a benchmark duration. This describes system capacity.
- Per-user output speed: the cadence at which one request receives generated tokens after generation begins. This is commonly expressed as the inverse of ITL.
- Prefill throughput: input tokens processed per second while the model handles a prompt. This describes prompt processing, not the rate at which the answer is generated.
Some benchmarks also combine input and output tokens. An input-plus-output figure is not comparable to output-only TPS unless the token scope is made consistent.
#1 Best Overall
Is TPS per user or for the whole system?
It can be either, depending on the benchmark. NVIDIA defines system TPS as the total output tokens per second across requests happening simultaneously. AIPerf, by contrast, reports per-user output throughput as the inverse of ITL: the steady cadence of tokens for an individual request. See the NVIDIA explanation of inference benchmark metrics and AIPerf metric definitions.
These numbers answer different questions. Aggregate TPS helps estimate how much work a serving system can handle. Per-user speed is closer to what one person experiences while reading a streamed response. Serving more requests at once can raise total system TPS while making each request slower. As load rises, throughput may eventually saturate at the available compute capacity; additional requests can then increase queueing and latency rather than increasing useful throughput.
Which latency measures should you read with TPS?
Time to first token (TTFT)
TTFT is the time from submitting a request until the first output token arrives. It captures the wait before a response starts and may include queueing, prompt processing, and network time. NVIDIA’s version 1.0.0 NIM metrics guide also describes tokenization and detokenization of the first token as part of TTFT. Longer prompts commonly take longer to prefill, so a model can have a high output TPS but still feel slow to start.
Recommended Free Tools
Inter-token latency (ITL) and time per output token (TPOT)
ITL, also called TPOT in some reporting, measures the average interval between generated tokens after the first token. In NVIDIA GenAI-Perf’s formula, ITL is (end-to-end latency − TTFT) ÷ (output tokens − 1). For a long answer, an individual request’s generation rate approaches 1 ÷ ITL. This is a decode-speed view for that request, not aggregate system throughput.
End-to-end request latency
End-to-end latency runs from sending the request until receiving its final token. It includes the initial wait, generation, and potentially queueing, batching, and network overhead. NVIDIA expresses it as TTFT plus generation time. This measure matters when the user needs the whole answer, rather than merely a quick start or a rapid token cadence.
Prefill throughput
Prefill throughput is the rate of processing input tokens in the prompt. AIPerf derives per-user prefill throughput for streaming responses by dividing input sequence length by TTFT. It is particularly relevant to workloads with long prompts, but it should not be mistaken for output generation speed.
Why can two TPS benchmarks disagree?
The timing window may be different
A tool’s benchmark boundaries affect the denominator. NVIDIA says GenAI-Perf calculates TPS over the interval from the first request to the final response, while LLMPerf divides by the full benchmark duration, which can include prompt generation, request preparation, and response storage. The results can therefore differ even for otherwise similar workloads. In its 2025 article, NVIDIA reports those LLMPerf overheads sometimes accounted for 33% of benchmark duration in a single-concurrency scenario; that is an observation from that scenario, not a universal overhead rate.
Token counting rules may differ
Benchmarks may use different tokenizers, chat templates, or definitions of output length. AIPerf distinguishes output tokens from output sequence length, which can include reasoning tokens. It also warns that a streamed first chunk can contain multiple tokens: treating that chunk as one token can inflate a derived per-user output-throughput value. MLCommons Endpoints says its official input and output sequence counts use a reference tokenizer and chat template. Its Endpoints documentation describes rules for that benchmark, not a universal convention.
Workload and load conditions may not match
Prompt and output lengths, concurrency, request arrival rate, batch limits, model, serving framework, hardware, precision or quantization, network conditions, and sampling settings can all affect results. Streaming behavior and treatment of empty responses also matter. A concurrency sweep can reveal trade-offs that one load point hides: total throughput may rise as more requests are served, while queueing and per-request latency worsen. When requests exceed an engine’s maximum batch size, queueing can increase even after throughput has saturated.
Reporting standards may use different conventions
MLCommons Endpoints v1.0 uses a steady-state window where applicable and presents P90 rather than P95. Those are conventions for its v1.0 results; they should not be assumed for other tools or benchmark reports.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to compare two tokens-per-second claims
Before treating one result as faster, check whether the following are aligned:
- Model and version, tokenizer, and serving framework.
- Hardware type and count, plus precision or quantization configuration.
- Prompt length and output length, including whether reasoning tokens count.
- Whether the figure is aggregate system TPS, per-user output speed, prefill throughput, or input-plus-output throughput.
- Concurrency or request-arrival pattern, batch limits, and streaming behavior, including how the first streamed chunk is counted.
- Warmup, ramp-up, steady-state, cooldown, and the precise measurement window.
- TTFT and a latency distribution, such as the reported percentile, alongside throughput.
- Whether the result is a reproducible measurement, a vendor-reported result, or a theoretical peak.
Then compare on the measure that fits the task: aggregate capacity for serving many simultaneous requests; TTFT and per-user ITL/TPOT for interactive use; or prefill throughput for long-prompt workloads. A peak system-wide TPS number is not a reliable proxy for the speed one user will feel.
What should a useful benchmark report include?
A report becomes interpretable when it states the numerator and denominator, token-counting method, workload, concurrency, and timing boundaries rather than publishing a lone TPS figure. For an interactive workload, pair throughput with TTFT, ITL or TPOT, and end-to-end latency. For a serving-capacity test, show how system TPS and latency change across load levels so readers can see where throughput stops improving and responsiveness degrades.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
