What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
LLM inference usually gets slower under load because incoming requests approach or exceed the model server’s workload-specific saturation point. Throughput then flattens while queues grow, so users wait longer for the first token, experience slower token streaming, or both. The correct fix depends on which phase is congested: measure time to first token (TTFT), per-output-token latency (TPOT or inter-token latency), end-to-end latency, and request states before changing the model or hardware.
Long prompts primarily increase prefill work and can delay the first token. Long generations occupy decode capacity and can slow the token cadence of other requests. Higher concurrency also increases competition for scheduler slots, GPU memory, and key-value (KV) cache space. Diagnose those causes separately, reproduce the same traffic shape in a benchmark, then tune scheduling, memory, model configuration, capacity, or topology against a defined tail-latency objective.
What “latency” means in an LLM service
A single end-to-end number hides where time is being spent. Interactive serving needs at least three measurements:
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →| Metric | What it measures | What a rise usually prompts you to investigate |
|---|---|---|
| Time to first token (TTFT) | Elapsed time from request admission until the first generated token is available. | Queueing or admission pressure, long-prompt prefill, scheduler contention, replica saturation, or KV-cache allocation delays. |
| TPOT / inter-token latency | Time between generated tokens after the first token. In the documented vLLM metric, TPOT is derived from end-to-end latency minus TTFT divided by output tokens after the first; requests producing at most one output token record zero. | Decode batching, too many active sequences, memory-bandwidth pressure, or interference from other work. |
| End-to-end latency | Total time from request start to completion. | The combined effect of admission, prefill, every decode step, queueing, and output length. |
Track distributions, not only averages. A p99 TTFT target can be missed even when mean TTFT looks healthy. Likewise, a stable TTFT with a deteriorating p99 inter-token latency points to a different problem than a queue that delays every request before generation starts.
#1 Best Overall
- [Local AI Inference & 70B Model Ready] Equipped with the AMD Ryzen 7 PRO 8845HS processor, NEXUS is engineered for heavy local AI workloads. With a full-size GPU bay, it runs 70B LLMs natively without an internet connection. Ideal for AI developers and tech enthusiasts who need private environment for coding and model testing.
- [132TB Mass Storage with ZFS Integrity] Features a hybrid storage architecture (3×NVMe + 4×3.5" HDD) supporting up to 132TB. Utilizing the enterprise-grade ZFS file system and ECC memory, it prevents data corruption and bit rot—a must-have for professional photographers and video editors safeguarding 4K/8K RAW footage.
- [OpenClaw-Driven Automation Workflow] The built-in OpenClaw execution layer allows complex automated tasks to be processed locally. Even when offline, your backup schedules and AI file organization continue seamlessly. Say goodbye to monthly cloud subscriptions and high latency.
- [Dual 10GbE & USB4 Ultra-Connectivity] Experience server-class speeds with dual 10GbE ports and a 40Gbps USB4 interface. It enables multi-user real-time collaboration on large project files directly from the NAS, ensuring zero-lag editing for creative studios and production teams.
- [Open-Source ZimaOS for Total Privacy] Running on the fully open-source ZimaOS, NEXUS ensures your data stays physically on-premise with no backdoors. It acts as a "Digital Fortress" for privacy-conscious families and small businesses who demand absolute data sovereignty.
In vLLM, the documented metrics include vllm:time_to_first_token_seconds, vllm:request_time_per_output_token_seconds, and counts of requests in RUNNING, WAITING, and SWAPPED states. Names and availability can change between installed versions, so check the metrics reference for the version you operate.
Why latency spikes as request rate rises
Saturation creates a queueing cliff
Every model-and-hardware configuration has a workload-specific point at which more arrivals stop producing more completed work. vLLM describes this as the inflection where a higher request rate no longer increases throughput but does increase latency. Before that point, spare compute or memory lets the scheduler absorb bursts. Near it, small increases in arrival rate or prompt length consume the remaining headroom. Beyond it, waiting requests accumulate and tail latency can rise sharply.
Do not use raw GPU utilization as proof of saturation. A device can show high utilization while the bottleneck is memory bandwidth, KV-cache capacity, a scheduler limit, or an imbalanced replica. Conversely, average utilization can look moderate when a burst periodically fills the queue. Establish saturation by varying offered load and observing where throughput flattens while latency accelerates.
Recommended Free Tools
Prefill and decode stress different resources
As Sarathi-Serve’s OSDI 2024 paper describes it, “The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of the output tokens, one-at-a-time.” Prefill handles the input sequence in a compute-intensive pass. Decode repeats a smaller step for each active sequence and benefits from batching across requests.
A long prompt therefore tends to increase TTFT, while a long output keeps a sequence in decode for more iterations and can affect the token cadence of its neighbors. When both phases share a scheduler, a large prefill can interrupt or delay decode work; many simultaneous decodes can in turn leave less capacity for new prefills. A traffic mix with short prompts and long answers behaves differently from one with long prompts and short answers, even at the same requests-per-second rate.
Batching trades throughput for responsiveness
Larger decode batches often improve aggregate tokens per second because the accelerator does more useful work per iteration. They also increase the amount of work a request may wait behind and can worsen tail latency. Interleaving prefill with decode improves utilization in some cases but can introduce additional interference. The best batch and concurrency limits are the ones that satisfy your service-level objectives (SLOs), not the settings that produce the highest benchmark throughput in isolation.
Memory pressure changes scheduling behavior
Each active sequence consumes KV-cache memory. Longer prompts, longer generations, and more concurrent sequences can exhaust available cache even when arithmetic capacity remains. A server may then keep requests waiting, swap state, or reduce the number of sequences processed together. Observe WAITING, RUNNING, and SWAPPED counts alongside latency to distinguish a compute-bound service from one constrained by memory or admission limits.
Free tools Windows power users keep installed
One-click scans. No signup required.
Use metrics to identify the failing phase
Start with request-level percentiles for TTFT, TPOT or inter-token latency, and end-to-end latency. Add request-state counts and the input and output token lengths for each request. The following patterns are useful hypotheses, not standalone proof:
| Observed pattern | Investigate first | Evidence to collect |
|---|---|---|
| TTFT rises while waiting requests increase | Admission pressure, scheduler backlog, prefill load, or saturated replicas. | Arrival rate, queue depth, prompt-token distribution, replica balance, and p50/p95/p99 TTFT. |
| TTFT is acceptable but TPOT or inter-token latency worsens | Decode batching, generation concurrency, memory bandwidth, or decode interference. | Active sequence count, output-token distribution, batch sizes, GPU memory behavior, and tail inter-token percentiles. |
| TTFT and TPOT both rise at high concurrency | Capacity at or beyond saturation, possibly combined with KV-cache pressure. | Offered request rate, throughput curve, waiting/running/swapped states, and cache usage. |
| Only long-prompt requests are slow | Prompt-length-dependent prefill cost rather than a global outage. | Input-token histograms, prefill timing by bucket, and whether short prompts retain normal TTFT. |
Detailed tracing can locate service-level delays, but vLLM warns that detailed trace collection may involve costly or blocking operations and can itself affect performance. Enable it briefly, on a controlled sample, and compare with tracing disabled.
Reproduce the spike with representative traffic
Do not tune from a single request, a constant-concurrency toy test, or an all-at-once throughput run. Reproduce the production shape as closely as possible.
- Define the SLO. Choose a concrete limit such as maximum p99 TTFT, maximum p99 end-to-end latency, or a token-cadence target. Also record goodput: the amount of work that completes within that SLO.
- Match the request mix. Use the observed distributions of prompt tokens, output tokens, stop reasons, and active sequences. Include short and long requests rather than testing only one fixed length.
- Match arrivals. Reproduce the average request rate, burstiness, concurrency limits, retries, and time-of-day peaks. A closed-loop test and an open-loop arrival test stress a server differently.
- Record the deployment. Capture model and precision, accelerator type and count, serving framework and version, tensor or pipeline parallel settings, batching limits, cache settings, and replica count.
- Ramp offered load. Increase request rate or concurrency in steps until throughput stops improving and latency rises. That curve identifies the saturation region for this workload.
- Compare percentiles and goodput. Save p50, p95, and p99 TTFT, TPOT, and end-to-end latency together with completed requests or tokens per second. A change that raises raw throughput but violates the p99 objective is not an interactive-serving improvement.
The vLLM project’s vllm bench serve is a server-oriented benchmark for simulating request arrivals and measuring serving behavior. Its latency and throughput modes represent different workload regimes; neither an isolated latency test nor an all-at-once throughput test should be treated as a production capacity result.
Fix scheduler and batching interference
Set limits from the latency objective
Reduce maximum concurrent sequences, batch size, or admission rate when queueing and tail latency are the problem. This can lower peak throughput, but it prevents a small number of long requests from monopolizing decode slots. Conversely, if TTFT is low and token cadence is healthy while the accelerator has headroom, a modest increase in batching may improve throughput without violating the SLO. Change one limit at a time and rerun the same benchmark.
Use continuous or chunked scheduling where supported
Serving engines that support continuous batching can admit and retire sequences between iterations instead of waiting for a static batch to finish. Chunked prefill divides a large prompt’s work so decode iterations are less likely to be stalled by one giant prefill. Availability, option names, and behavior depend on the serving engine and version; enable these features only after confirming compatibility with the deployed model and measuring their effect on TTFT and tail TPOT.
The Sarathi-Serve OSDI 2024 paper presents chunked-prefill and stall-free scheduling as one approach to the prefill/decode trade-off. Its reported gains are experimental results for specific models, GPUs, workloads, and service constraints, not a guaranteed improvement for another deployment.
Protect interactive traffic from pathological requests
- Apply maximum input and output token limits that are consistent with the product contract.
- Separate latency-sensitive traffic from batch or offline generation when the serving stack permits it.
- Use admission control or per-tenant concurrency limits so one caller cannot fill every active sequence.
- Measure retries: a timeout-driven retry storm can raise offered load precisely when the server is saturated.
Evaluate memory and model optimizations
Quantization
Quantization stores model weights at lower precision, reducing memory requirements and sometimes improving latency or throughput. The outcome depends on the quantization method, model architecture, accelerator, and serving framework. Google Cloud’s GKE guidance also notes a possible accuracy trade-off and framework-specific compatibility. Compare answer quality on representative evaluation sets whenever precision changes; a faster model that fails quality gates is not a valid fix.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Tensor parallelism and memory configuration
Tensor parallelism can spread a model across multiple accelerators when one device lacks memory or when the implementation benefits from parallel execution. Communication overhead, interconnect bandwidth, partitioning support, and the model’s shape determine whether it helps. Memory optimizations can increase the number of resident sequences, but they may also alter stability or scheduling behavior. Benchmark the exact model, precision, accelerator count, and parallelism configuration rather than assuming that adding devices reduces per-request latency.
Check cache and length assumptions
Inspect KV-cache usage under the longest realistic prompts and generations, not just the average. A configuration that is stable at median lengths may begin swapping or rejecting work at the tail. If the product permits it, shortening prompts, limiting maximum generation length, or reusing safely cacheable context can reduce both prefill work and resident-state pressure; validate any caching change against correctness and isolation requirements.
Scale from measured saturation
Use the throughput-versus-offered-load curve to decide when to add replicas or accelerators. A useful scaling signal combines latency percentiles, waiting and swapped request counts, completed tokens, and the request-length mix. There is no universal request-rate threshold: the saturation point changes with model, precision, prompt lengths, output lengths, hardware, and SLO.
Scale-out helps when compute or memory capacity is the constraint and new capacity becomes available before the burst has already timed out. Before adding replicas, verify:
- Routing balance: requests are distributed evenly and long-lived streams are not pinned disproportionately to one replica.
- Model loading: loading time, image size, and weight transfer do not create a second queue during rollout or autoscaling.
- Memory fit: each replica can hold the model plus the intended KV-cache working set.
- Startup behavior: the deployment has measured warm-up and readiness times; the sources reviewed do not establish a universal cold-start duration.
- Network and orchestration limits: service discovery, ingress, and accelerator scheduling can deliver the additional capacity.
When the burst is short, admission control and a bounded queue may protect the SLO better than launching a replica that will not be ready in time. When demand is sustained, capacity planning should keep normal operation below the measured inflection point rather than treating maximum theoretical throughput as usable capacity.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.When prefill/decode disaggregation makes sense
Disaggregated serving places prefill and decode on separate worker groups. The goal is to stop a large prefill from disrupting the decode cadence of active streams. The trade-off is transferring the prompt’s KV cache between workers. That transfer can add TTFT and become the new bottleneck.
A vLLM project report from 2026 describes a workload-specific L40S test in which collocated p99 inter-token latency increased from 23 ms to 169 ms at 0.4 requests per second, while the disaggregated setup reported 29 ms at that rate. Those figures apply only to the report’s named model, two L40S GPUs, prompt and output sizes, request-arrival setup, and other configuration details; they are not portable promises.
Evaluate this topology only after measuring:
- KV-cache transfer bandwidth and transfer time on the actual network and hardware;
- TTFT, including transfer overhead;
- p95 and p99 inter-token latency during mixed prefill and decode load;
- goodput under the same SLO and request mix;
- failure handling when a prefill or decode worker is unavailable.
Collocated serving remains appropriate for low or bursty traffic and for workloads that are primarily TTFT-bound. Separation is more attractive when sustained mixed traffic causes decode-tail interference and the transfer path is demonstrably fast enough.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallWhat published capacity figures do—and do not—prove
The Sarathi-Serve authors reported the following capacity differences in their 2024 OSDI experiments:
| Experiment | Reported result | How to interpret it |
|---|---|---|
| Mistral-7B on one A100 GPU | 2.6× higher serving capacity than vLLM in the paper’s experiment. | Configuration-specific evidence for the tested model, GPU, workload, and service constraints. |
| Yi-34B on two A100 GPUs | Up to 3.7× higher serving capacity than vLLM in the paper’s experiment. | Not a general multiplier for other models or hardware. |
| Falcon-180B with pipeline parallelism | Up to 5.6× gain in end-to-end serving capacity in the paper’s experiment. | Applies to the paper’s pipeline-parallel setup and measured workload. |
Use such results to understand why scheduling design matters, not to forecast your own capacity. Reproduce the relevant traffic and hardware before using any published number in a capacity plan.
A practical decision framework
| If your primary failure is… | First actions | Trade-offs to verify |
|---|---|---|
| High TTFT with a growing waiting queue | Measure prompt lengths, prefill time, admission limits, and replica saturation; reduce offered concurrency or improve prefill scheduling. | Lower peak throughput, fairness between short and long prompts, and p99 TTFT. |
| High TPOT with acceptable TTFT | Inspect decode batch size, active sequences, output lengths, memory bandwidth, and prefill interference. | Tokens per second versus streaming smoothness and p99 inter-token latency. |
| Both TTFT and TPOT degrade near a load threshold | Find the saturation point, add capacity if startup and routing permit, and reduce KV-cache pressure. | Hardware cost, model-loading delay, and whether the new replica receives balanced traffic. |
| Memory pressure or swapped requests | Review maximum lengths and concurrency; evaluate quantization or memory optimizations. | Accuracy, framework compatibility, stability, and cache behavior. |
| Decode tails disrupted by sustained long prefills | Benchmark chunked scheduling or prefill/decode disaggregation. | Implementation complexity, transfer bandwidth, and possible TTFT increase. |
Runbook: a safe change-and-verify loop
- Capture a baseline for the production-like workload, including p50/p95/p99 TTFT, TPOT, end-to-end latency, throughput, goodput, request states, and token-length distributions.
- Classify the symptom as prefill/admission, decode, memory, or total-capacity pressure. Treat the classification as a hypothesis and confirm it with a load ramp.
- Apply one change: a scheduler limit, batching policy, length limit, precision, parallelism setting, replica count, or topology change.
- Repeat the identical benchmark and configuration capture. Do not compare a latency-mode result with a throughput-mode result or change the request mix between runs.
- Accept the change only if the target tail-latency SLO and quality requirements hold while throughput or goodput improves. Revert changes that merely improve an average or peak throughput number while worsening interactive tails.
- Keep a workload-specific saturation curve and update it after model, hardware, framework, or traffic-mix changes.
The central rule is simple: measure the phase that hurts the user, then choose the control that acts on that phase. Queueing before the first token, interference between generated tokens, and total capacity exhaustion can look similar in a dashboard but require different fixes.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
