Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
The right way to measure LLM speed depends on what users experience. For a streaming chat interface, measure time to first token (TTFT) and the gaps between streamed tokens (inter-token latency, or ITL). For a synchronous API, measure end-to-end latency to the final token. For offline batch work, measure aggregate output-token throughput. Always publish the workload, hardware, software, cache state and percentile distributions with the number; an unqualified “tokens per second” figure is not a reproducible benchmark.
This guide defines each metric, shows the formulas, gives a client-side measurement procedure, explains prefill versus decode, and provides a fair-comparison checklist you can reuse across models and serving engines.
What “LLM inference speed” should mean
Inference speed is not one universal number. A server can generate many tokens per second in aggregate while an individual user waits several seconds for the first token. Conversely, a single request can decode quickly while a busy service has poor tail latency because requests queue behind one another.
State the exact quantity whenever you report speed:
#1 Best Overall
- Output-token throughput: generated output tokens divided by the benchmark interval. This is the most useful aggregate capacity number for many serving tests.
- Total-token throughput: input plus output tokens divided by the interval. Compare this only with another result that counts tokens the same way.
- Per-request decode speed: approximately
1 / TPOTtokens per second after the first token. It describes one request’s steady-state generation, not the server’s aggregate capacity. - Request throughput: completed requests per second, which matters when each request has a known or controlled token length.
A complete report says which definition is used, whether the clock is measured at the client or server, and whether tokens are counted at input, output, or both.
The four latency metrics you need
| Metric | Definition | What it reveals | Best use |
|---|---|---|---|
| TTFT (time to first token) | Elapsed time from request submission until the first streamed output is received. | Startup delay, including prompt processing, queueing and (at the client boundary) network time. | Interactive streaming responsiveness. |
| ITL (inter-token latency) | Wall-clock gap between consecutive streamed outputs. | Smoothness of the stream. With one token per event, it approximates per-token decode delay. | Detecting pauses and jitter during streaming. |
| TPOT (time per output token) | (end-to-end latency - TTFT) / (output tokens - 1) for one request. |
Request-level average decode time after the first token. | Estimating per-request decode speed. |
| End-to-end latency | Time from request start until the final output token or event. | Total wait for a completed response. | Synchronous APIs and completion-time SLOs. |
TPOT excludes the first token by design. If a response has one output token, the denominator is zero and TPOT is undefined; report TTFT and end-to-end latency instead. ITL and TPOT can diverge when a transport event bundles several tokens, when speculative decoding is enabled, or when the server emits unevenly sized chunks. Metric terminology is not standardized across benchmarking tools, as the vLLM Project notes, so publish the formula and measurement point alongside every label.
Why prefill and decode behave differently
Inference has two practical phases. During prefill, the engine reads the complete user prompt and computes the state needed to begin generation. During decode, it generates the response one token at a time while reusing that state. NVIDIA describes prefill as generally compute-bound for reasonable prompt lengths and decode as usually memory-bound.
Prompt length mainly affects prefill
A longer prompt usually increases TTFT because more input tokens must be processed before the first output can be produced. Include the system prompt, conversation history and any retrieved or multimodal text in your input-token count. A benchmark that reports only the user’s last message understates the actual prefill workload.
Accumulated context affects decode
Every generated token extends the context that the engine carries forward. Long conversations and long outputs can therefore reduce decode efficiency, increase memory pressure and raise ITL even when the prompt itself is unchanged. Measure prompt and output lengths as distributions rather than relying on one “average” request.
Do not infer one phase from the other
A system can have excellent steady-state decode speed but a slow first token, or a fast first token but unstable token gaps under contention. Report TTFT and decode metrics separately instead of collapsing both phases into one tokens-per-second figure.
Choose the headline metric for your workload
| Workload | Lead with | Also report | Reason |
|---|---|---|---|
| Interactive streaming chat | TTFT p50/p95/p99 and ITL or TPOT p50/p95/p99 | End-to-end latency and output-token throughput | Users notice when the response starts late or stalls between tokens. |
| Synchronous completion API | End-to-end latency percentiles | TTFT, output-token throughput and output length | The caller waits for the complete response. |
| Offline batch generation | Aggregate output-token throughput | Requests per second, total-token throughput, utilization and queue time | Capacity and job completion time dominate user experience. |
| Capacity planning | Throughput at the target request rate | Prompt/output distributions, concurrency, peak requests per second and latency SLO percentiles | NVIDIA’s checklist requires these workload inputs to size a service realistically. |
As human-reading context, NVIDIA’s 2024 material cites 90 ms per token as fast reading and 200 ms per token as average reading. These figures are intuition for perceived pacing, not performance targets or substitutes for a service-level objective.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11A reproducible measurement procedure
1. Declare the question before running a test
Write one sentence such as “I need TTFT and p95 ITL for a streaming assistant at 8 concurrent users,” or “I need output-token throughput for a batch job.” This prevents an aggregate capacity result from being presented as a single-user experience.
Rank #2
2. Freeze the system configuration
Record all variables that can change performance:
- Model identifier and exact revision or commit.
- Tokenizer and vocabulary version.
- Quantization or numerical precision.
- GPU or CPU model, memory, driver and firmware versions.
- Serving-engine version, scheduler settings, context limit and number of replicas.
- Sampling parameters such as temperature, top-p, top-k, repetition penalties, maximum output tokens and stop sequences.
- Endpoint protocol, streaming setting and transport (for example, HTTP streaming).
- Whether prefix or prompt caching is enabled, and the cache state at test start.
If any item changes between runs, treat the results as a different experiment.
3. Define a representative workload
Specify the distribution, not just an average: prompt-token lengths, output-token lengths, system prompts, conversation turns, stop conditions, tool calls and multimodal inputs. Record request arrival rate and concurrency. For synthetic data, use a fixed random seed; for a real corpus, keep a versioned dataset and document redaction or sampling rules.
4. Timestamp at the client boundary
Capture a monotonic timestamp for request submission, every streamed event and the final event. Also record HTTP errors, timeouts, cancellations and rejected requests. Client-boundary timing includes network and queue delay that a user experiences. Server-side timers can be useful for diagnosing compute, but they are not interchangeable with client-observed TTFT.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
5. Warm up, then collect enough samples
Discard startup and warm-up requests so model loading, kernel compilation and memory allocation do not dominate the result. Run enough successful requests for p50, p95 and p99 to stabilize; report the request count and success rate. A handful of samples cannot support a reliable tail percentile.
6. Control cache state
Prefix-cache hits can make TTFT and throughput look artificially high. For an independent comparison, restart or reset caches between runs, or report separate cache-hit and cache-miss scenarios. Never mix unknown cache states in one average.
7. Compute the metrics
For each request, calculate:
TTFT = first_event_time - request_send_timeITL[i] = event_time[i] - event_time[i-1]for each consecutive streamed eventend_to_end = final_event_time - request_send_timeTPOT = (end_to_end - TTFT) / (output_tokens - 1)when output tokens exceed one
For the whole interval, calculate output-token throughput = total_output_tokens / benchmark_seconds, total-token throughput = (total_input_tokens + total_output_tokens) / benchmark_seconds, and request throughput = successful_requests / benchmark_seconds. Keep per-request values for percentile calculations; do not derive p95 TPOT by dividing a p95 latency by an average token count.
8. Repeat across concurrency or request rate
Single-user decode speed and saturated multi-user throughput answer different questions. Sweep concurrency or arrival rate and plot throughput against TTFT, TPOT or end-to-end latency. The useful operating point is usually near the highest throughput that still meets your latency SLO, not the absolute maximum throughput at an unusable queue depth.
Free tools Windows power users keep installed
One-click scans. No signup required.
Minimal client-side instrumentation
The following Python-like example shows the required timestamps. Adapt the streaming iterator to your SDK; the important part is the measurement boundary and the per-event clock.
send = monotonic()
first = None
last = None
event_times = []
output_tokens = 0
for event in stream_response:
now = monotonic()
event_times.append(now)
if first is None:
first = now
last = now
output_tokens += count_tokens(event.text_delta)
end_to_end = last - send
TTFT = first - send
ITL = [event_times[i] - event_times[i-1]
for i in range(1, len(event_times))]
TPOT = ((end_to_end - TTFT) / (output_tokens - 1)
if output_tokens > 1 else None)Use a monotonic clock rather than wall-clock time, because wall clocks can jump when synchronized. Count tokens with the same tokenizer used by the model. If an event contains multiple tokens, retain both event-level ITL and token-level counts; do not call the event gap “per-token latency” without explaining the bundling.
Using a benchmark tool without losing comparability
The official vLLM command-line benchmark, vllm bench serve, reports successful requests, benchmark duration, input and generated tokens, request throughput, output-token throughput, total-token throughput, TTFT, TPOT and ITL. It accepts custom JSONL datasets, deterministic synthetic data, saved detailed results and selectable percentiles.
Before comparing its output with another framework, verify the tool’s formulas and timestamp location. One tool may start its clock at client submission while another starts after a server receives the request. One may count only generated tokens while another reports input plus output. Run vllm bench serve --help for the installed version’s exact dataset, concurrency, percentile and output-file options, then record those options in your benchmark log.
Recommended Free Tools
How to make two systems genuinely comparable
Use this checklist for every A-versus-B comparison:
| Axis | Requirement | Why it matters |
|---|---|---|
| Model | Same model family, revision and tokenizer | Architecture, weights or tokenization changes alter work per token. |
| Precision | Same numerical precision or quantization | Memory traffic and arithmetic cost can change substantially. |
| Hardware | Same accelerator or CPU type, memory and driver class | Different devices invalidate a direct speed claim. |
| Engine | Record engine version and scheduler settings | Kernel, batching and scheduling changes affect both latency and throughput. |
| Workload | Same prompt/output-length distributions, system prompts and stop rules | Prefill and decode costs depend on token counts. |
| Load | Same concurrency, request rate and replica count | Queueing and batching determine tail latency. |
| Protocol | Same endpoint, streaming mode and event format | Streaming and chunking change TTFT and ITL observations. |
| Cache | Same warm or cold state, or separate hit/miss results | Prefix reuse can dramatically reduce prompt work. |
| Timing boundary | Both client-side or both server-side, with definitions | Network and queue time may be included on one side only. |
| Accounting | Same input/output token convention and percentile method | Different denominators produce incomparable “tokens per second.” |
| Errors | Publish success, timeout, cancellation and rejection policy | Dropping failed slow requests makes results look better. |
Report distributions, not just averages
At minimum, publish request count, success rate, benchmark duration, input and output token totals, request throughput, output-token throughput, total-token throughput, TTFT p50/p95/p99, TPOT p50/p95/p99, ITL p50/p95/p99 and end-to-end latency p50/p95/p99. Include the percentile definition and whether it is calculated over requests or individual token gaps.
The mean can hide queueing and contention. For example, most requests may start quickly while a small group waits behind a batch and produces a poor p99. A streaming service should expose ITL tails because a few long pauses are perceptible even when average ITL is low. For capacity planning, pair every throughput point with its latency percentiles and offered load.
Worked figures and how to interpret them
NVIDIA’s 2024 worked example uses 3,500 input words, 500 output words, a three-second first-token limit and a peak of 31 requests per second as planning inputs. The same example reports 2.4 requests per second per DGX H100 system, 2,606 ms first-token latency and 21.4 ms per generated token. Those values describe that particular model, hardware, workload and configuration; they are not universal benchmarks or guarantees for another system.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →At 21.4 ms per generated token, the approximate single-request decode rate after the first token is 1 / 0.0214 ≈ 46.7 tokens per second. That derived rate must not be confused with the example’s aggregate 2.4 requests-per-second capacity or with its 2,606 ms TTFT. A long first-token delay and a fast subsequent stream can coexist.
Diagnose surprising results
TTFT is high but ITL is normal
Check prompt length, queue time, cold model loading, prefix-cache misses and network setup. The pattern usually points to prefill or scheduling rather than decode kernels.
ITL or TPOT worsens as output grows
Inspect accumulated context, memory pressure, paging or eviction, and whether the scheduler is mixing long and short requests. Compare runs with controlled output lengths.
Average speed is high but p99 is unacceptable
Increase the concurrency sweep, inspect queue depth and rejected or timed-out requests, and report the tail rather than optimizing only the mean. Saturation often appears first in p95 or p99.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesITL is much lower or higher than TPOT
Look at event framing. Multi-token chunks, speculative decoding, buffering in the client library or network coalescing can make event-to-event gaps differ from request-level average decode time. Report tokens per event and both metrics.
Two tools disagree on “TPOT”
Read each tool’s formula and timing point. Names are not standardized; one implementation may include the first token, count only successful requests, or use server timestamps. Recompute from raw timestamps when possible.
Throughput rises after enabling a cache
Separate cache-hit and cache-miss tests and disclose the hit rate. A cached benchmark answers a different question from a cold-prefix benchmark.
A practical benchmark record
Keep one machine-readable record per run containing:
- Date and time, model revision, tokenizer, precision, hardware and driver.
- Serving-engine version, replica count, context limit and scheduler configuration.
- Dataset identifier or synthetic-data seed, plus prompt and output token distributions.
- Sampling, stop, streaming and cache settings.
- Concurrency or request-rate sweep, warm-up count and measured duration.
- Successful, failed, timed-out and rejected request counts.
- Input, output and total token counts.
- Per-request TTFT, TPOT and end-to-end values, plus per-event ITL values.
- Aggregation method and p50/p95/p99 results.
Saving raw per-request timings makes later percentile or accounting changes possible without rerunning an expensive test.
How to present a result so readers can use it
Lead with a sentence that fixes the meaning: “At 16 concurrent streaming requests, output-token throughput was X, TTFT was Y ms p50 and Z ms p95, and TPOT was A ms p50.” Immediately follow it with the model revision, hardware, precision, prompt/output distributions, cache state and measurement boundary. A table is useful for multiple load points, but never omit the conditions that produced each row.
If the audience is choosing an interactive system, show a latency-throughput curve and identify the point that meets the SLO. If the audience is scheduling batch work, show output tokens per second and projected completion time at the expected queue depth. In both cases, preserve the distinction between a request’s decode speed and the service’s aggregate capacity.
Frequently Asked Questions
Should I use tokens per second or milliseconds per token?
Use the unit that matches the decision, then provide the conversion. Per-request decode speed is approximately 1 / TPOT; report TTFT and percentile latency separately so the conversion does not hide startup delay or tail behavior.
Do input tokens count in tokens-per-second?
Only when you explicitly use total-token throughput. Output-token throughput counts generated tokens; total-token throughput counts input plus output. Never compare the two conventions as if they were the same metric.
Why can a benchmark with streaming disabled look faster?
Without streaming, the client cannot observe TTFT or ITL and may measure only one final response event. Buffering and endpoint behavior change the measurement, so use the same streaming mode when comparing interactive systems.
How many requests are enough for p99?
There is no universal count, but p99 is unstable with a small sample. Run until the percentile stops moving materially, publish the request count and success rate, and avoid presenting a tail percentile from only a handful of requests.
What should a capacity-planning request specify?
Specify peak requests per second, concurrency, prompt and output length distributions, model and hardware configuration, cache assumptions and the latency SLO. Those inputs determine whether a throughput figure is relevant to your deployment.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
