The Tool Desk

Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.

iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more

There is no universal “good” tokens-per-second (TPS) score for a large language model. TPS can mean the speed of one response or the combined output of many concurrent requests, and neither number says much without the prompt, model, latency, and measurement method. To benchmark fairly, define which decision the test should inform, use a fixed workload, measure both responsiveness and throughput, and report the conditions alongside the results.

What tokens per second measures—and what it leaves out

Tokens per second is a rate, but benchmark tools do not all define the rate the same way. Check whether a reported figure counts generated output tokens alone or combines input and output tokens; whether timing starts before or after the first token; and whether the result represents one request or multiple requests together. Those choices can produce numbers that look comparable but answer different questions. NVIDIA notes that benchmarking tools can use different metric definitions in its guide to LLM inference benchmarking concepts.

# Preview Product Price
1 Inference Jones, Level 1 Inference Jones, Level 1 $11.99 Buy on Amazon
2 LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production Systems — Real Benchmarks, Python Code and Complete Code Repository for Engineers at Scale LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production... $29.99 Buy on Amazon
3 Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and speculative decoding (Scaling AI Systems Series) Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and... $9.99 Buy on Amazon
4 AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work $12.99 Buy on Amazon
5 LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and Prove Your Model Got Better — LoRA, QLoRA, SFT, DPO and GRPO LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and... $9.99 Buy on Amazon

For example, Ollama describes its per-request TPS as output tokens generated per second after the first token, excluding the initial wait. That is a project-specific methodology, not a universal standard; its methodology explanation is useful for understanding what its figure represents.

Recommended: Fix Windows Errors and Clear Junk Files in Minutes - Free Scan →Free scan · no reinstallRecommended: Crashes or Glitches? A Free Driver Scan Usually Finds the Culprit →Free scan · under a minuteRecommended: PC Feels Slow? A Free Scan Shows What's Dragging Windows Down →Free scan · Windows 10 & 11
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Per-request output TPS: the generation pace of a single response stream. It does not include startup delay under the Ollama definition and does not establish how many users a service can support.
  • Aggregate output throughput: all output tokens produced per second across concurrent requests. This describes system capacity under a stated workload, not the pace experienced by each individual user.
  • Input-plus-output rates: some tools may count both prompt and generated tokens. Such a figure should not be treated as output TPS unless the counting rule is clear.

Always keep the unit visible: TPS means tokens per second. Latency metrics such as TTFT and TPOT are usually given in milliseconds or seconds; converting a token interval into a rate requires taking its reciprocal and stating which interval is being converted.

#1 Best Overall
Inference Jones, Level 1
Inference Jones, Level 1
$11.99
Buy on Amazon

Which speed metrics matter to a user?

An interactive response has an initial wait followed by a stream of generated tokens. Looking at only one phase can obscure the experience: a model may begin quickly but generate slowly, or take longer to begin and then stream rapidly.

  • Time to first token (TTFT): elapsed time until the first content token arrives. It can include queuing, prompt processing, and network time, depending on where and how it is measured. NVIDIA describes a client-side TTFT measure that includes queuing, prefill, and network latency. Its authors define TTFT as “the time it takes to process the prompt and generate the first token” in the NVIDIA benchmarking concepts article.
  • Time per output token (TPOT) or inter-token latency (ITL): the average interval between generated tokens after the first. Definitions vary by tool. NVIDIA’s GenAI-Perf definition excludes TTFT and divides generation time by the output-token count minus one.
  • End-to-end latency: time from sending the request until the final token is received. It includes the initial wait and generation, though exact treatment of queuing and transport depends on the measurement method.

For one complete answer, a useful mental model is: total response time consists of the wait for the first token plus the time spent generating the remaining output. A prompt’s input tokens are processed during prefill; the model then generates output autoregressively, one token at a time. Longer prompts can extend prefill and TTFT, while longer outputs generally add generation time. Databricks explains this distinction and the latency-throughput trade-off in its endpoint benchmarking guidance.

Rank #2
Sale
LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production Systems — Real Benchmarks, Python Code and Complete Code Repository for Engineers at Scale
LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production Systems — Real Benchmarks, Python Code and Complete Code Repository for Engineers at Scale
$29.99
Buy on Amazon

How many tokens per second is a good speed for an LLM?

There is no evidence-backed universal threshold. A useful target depends on the model, prompt and output lengths, whether the system serves one person or many, and the latency the application can tolerate. A TPS result without those conditions is not enough to decide whether a system is fast for your purpose.

What’s actually slowing this PC down?

Pick the symptom - the matching free tool is one click away.

Junk files & broken settingsFree fix →System errors & instabilityFree fix →Crashes & glitchesFree fix →Outdated driversFree fix →
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
  • Interactive use: prioritize TTFT, TPOT or ITL, and full response time. A high generation rate does not compensate for an unacceptable initial wait.
  • Batch processing: aggregate output tokens per second may matter more than the speed of any one response, provided the resulting completion times meet the job’s requirements.
  • User-facing service: measure throughput while enforcing a latency budget. Databricks recommends maximizing throughput within the application’s latency budget, rather than treating peak throughput as the goal.

To evaluate a particular service, define an acceptable wait and completion time for its real users, then test whether it meets those requirements under the expected workload. Google Cloud’s accelerator benchmarking guidance likewise frames inference comparisons around fixed workloads and latency-constrained throughput.

Rank #3
Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and speculative decoding (Scaling AI Systems Series)
Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and speculative decoding (Scaling AI Systems Series)
$9.99
Buy on Amazon

How to benchmark LLM inference speed reproducibly

  1. Decide what the result will inform. Choose whether you are evaluating interactive responsiveness, sizing an endpoint, comparing local accelerators, or estimating batch capacity. That decision determines which metrics and latency limits matter.
  2. Fix a representative workload. Use the same prompt set and specify input and output token lengths or their distributions. For a fair comparison, hold the model and version, tokenizer, quantization or precision, serving stack, streaming mode, and generation settings constant. Prompt length affects prefill and initial wait; output length affects generation duration.
  3. Warm up and repeat the test. Record the tool and methodology, warm-up approach, number of runs, and whether each reported statistic is a mean, median, or percentile. NVIDIA’s NIM latency-throughput benchmarking guide organizes testing around warm-up, workload sweeps, and analysis; use the documentation for the exact tool and version to determine command options.
  4. Measure a single stream and a concurrency sweep. A single request characterizes one stream’s generation pace. Increase concurrent requests to observe aggregate throughput, latency, and queuing. As parallel work rises, throughput may improve while response latency also rises; eventually, a provisioned-capacity limit can constrain further gains.
  5. Record a complete metric set. Include per-request output TPS or TPOT, TTFT, end-to-end latency, aggregate output throughput, concurrency, and success or error rate. Report p50 and a tail percentile such as p95 or p99 when there are enough observations to make that percentile meaningful.
  6. Find the operating point that meets the service constraint. For an interactive service, report the throughput achieved before the chosen latency target is exceeded, not just the highest raw throughput observed. Google Cloud describes increasing concurrency until the p99 latency service-level objective is violated, then recording sustained throughput in its benchmarking guidance.
  7. State what the measurement includes. Disclose whether results came from an independent test, a vendor-published benchmark, or another named tool’s methodology. For hosted services, network path, queuing, and load can affect client-side measurements. A single run or a vendor headline is not a universal model or hardware specification.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Support on Ko-Fi

What to include when publishing a TPS result

  • Model name and version; tokenizer, precision or quantization, serving stack, and generation settings.
  • Prompt workload, input and output token lengths or distributions, and whether requests streamed output.
  • Metric definitions: counted tokens, timing start and stop, and whether the figure is per request or aggregated.
  • Concurrency and test duration, plus the warm-up and repeated-run method.
  • TTFT, TPOT or ITL, end-to-end latency, aggregate output throughput, and error or success rate.
  • Summary statistics, including p50 and an adequately sampled tail percentile where available.
  • For service comparisons, the latency target and the throughput achieved while meeting it.
  • Measurement scope: hardware and configuration for local tests, or relevant endpoint and network conditions for hosted tests; identify the source of the result.

When comparing systems, align the workload and report responsiveness, capacity, and tail behavior separately. If efficiency or cost matters, normalize it to a clearly stated scope, such as performance per accelerator or per dollar, and disclose the hardware and pricing basis. A speed result alone does not establish model quality, and unlike prompts or output lengths do not make a fair comparison.

Quick Recap

Bestseller No. 1
Inference Jones, Level 1
Inference Jones, Level 1
$11.99
SaleBestseller No. 2
LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production Systems — Real Benchmarks, Python Code and Complete Code Repository for Engineers at Scale
LLM Inference Engineering Handbook: Crush API Costs, Cut Latency and Build Reliable Production Systems — Real Benchmarks, Python Code and Complete Code Repository for Engineers at Scale
$29.99
Bestseller No. 3
Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and speculative decoding (Scaling AI Systems Series)
Inference at Full Throttle: LLM serving performance with vLLM, quantization, KV cache tuning and speculative decoding (Scaling AI Systems Series)
$9.99
Bestseller No. 4
AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work
AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work
$12.99
Bestseller No. 5
LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and Prove Your Model Got Better — LoRA, QLoRA, SFT, DPO and GRPO
LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and Prove Your Model Got Better — LoRA, QLoRA, SFT, DPO and GRPO
$9.99
Best Value
LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and Prove Your Model Got Better — LoRA, QLoRA, SFT, DPO and GRPO
LLM Fine-Tuning That Works: Decide When to Fine-Tune, Build the Right Data, Train on One GPU, and Prove Your Model Got Better — LoRA, QLoRA, SFT, DPO and GRPO
$9.99
Buy on Amazon
Rank #4
AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work
AI Decision Fatigue: Stop Chasing Benchmarks and Start Matching Models to Your Actual Work
$12.99
Buy on Amazon

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.