Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A local AI model can feel slow for several different reasons: reading the prompt (prefill), producing tokens (generation), waiting for the first token, or handling several requests at once. Measure the slow phase first. Then verify that your runtime was built for the intended accelerator, confirm from startup diagnostics that offload is active, tune CPU threads experimentally, and reduce memory pressure only as far as your workload allows.
The examples below use llama.cpp where command-line behavior is discussed. Other local runtimes expose different controls, so treat the workflow as portable but not every flag as universal.
What “slow” means in local inference
Two runs can have the same total response time while having completely different problems. A long document may take a while to ingest but then generate quickly. A short prompt may produce its first token promptly and then crawl. A single chat can be acceptable while four simultaneous requests collapse the machine’s throughput.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Symptom | What to measure | Likely area to investigate first |
|---|---|---|
| Long wait before the first token | Prompt-processing time or time to first token | Batching, BLAS/prefill support, prompt length, and memory pressure |
| First token arrives quickly, then output is slow | Generated tokens per second | CPU-thread setting, active GPU offload, model size, and thermal or memory limits |
| Short prompts are fine but long conversations bog down | Prefill time and context length | Context allocation, prompt size, and available RAM or VRAM |
| One request is fine but concurrent use is slow | Aggregate tokens per second and latency per request at a fixed concurrency | Memory headroom, scheduling, and whether the backend supports the workload efficiently |
| Changing a setting has no visible effect | The same phase-specific metric before and after the change | Whether the setting affects that phase at all, and whether the intended backend is actually active |
Do not use a single “tokens per second” number as a complete diagnosis. Prompt ingestion and token generation are separate workloads and often respond to different settings.
#1 Best Overall
Measure before changing hardware or settings
Record the complete test setup
Write down the model name, parameter count, file format, quantization, runtime and version or build, operating system, CPU model, GPU model, available system memory, available GPU memory, context length, prompt size, thread count, GPU-layer setting, and any batching or backend options. Also note whether the run is a cold start or a repeated request with data already cached.
A result without this information is not a transferable target. A 30B four-bit model on a machine with 48 GB of VRAM is a fundamentally different test from the same nominal model on an integrated GPU and shared system memory.
Measure prompt and generation phases separately
Use the runtime’s timing output, if available, to capture prompt-processing rate and generation rate independently. Record time to first token as well as steady-state generation. If the runtime reports prompt tokens per second and generated tokens per second, keep both values; improving one does not prove that the other improved.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsRepeat comparable runs
Run the same prompt or benchmark more than once, allow the machine to reach a similar temperature, and keep background workloads stable. Change one variable at a time. A benchmark that changes the model, context, thread count, and backend together cannot tell you which change mattered.
The llama.cpp llama-bench utility supports repeated tests and structured output containing the machine, backend, model, configuration, and timing or throughput fields. Use those records to compare like with like. Its documentation includes a sample test timestamp of 2025-04-24; that timestamp identifies an example record, not a current performance result or a recommended speed.
Confirm that acceleration is really active
Read llama.cpp startup diagnostics
For a llama.cpp build intended to use CUDA, inspect the startup log rather than trusting a GPU-layer limit in your command. The troubleshooting documentation describes diagnostics that report layers moved to the GPU and the VRAM used by the model. Those messages are evidence that the selected backend is doing work. A configured value for the number of GPU layers, by itself, is not proof of useful GPU execution.
If the expected offload diagnostics do not appear, stop tuning threads and batch sizes. Check that the executable was built with the required CUDA support, that the runtime selected the intended device, that the device is visible to the process, and that the model layers fit the available accelerator memory. A CPU-only fallback can make every later optimization look ineffective.
Check the backend that matches your hardware
llama.cpp documents CUDA support for NVIDIA GPUs, Metal support for Mac GPUs, and SYCL support for Intel GPU classes. The build steps, driver requirements, and supported model paths differ by backend. Select the documented backend for your operating system and device, then verify its startup messages on every machine rather than assuming that a binary built elsewhere has the same capabilities.
OpenVINO is a separate backend path. Its documentation lists Intel CPUs, GPUs, and NPUs, with validation on specified Intel Core Ultra Series 1 and Series 2 systems. The page describes optimization and broader model or quantization coverage as work in progress, so compatibility must be checked for the exact model, device, build, and driver.
Tune CPU threads as an experiment
More CPU threads can make generation slower. Thread coordination, memory traffic, and contention with the accelerator can outweigh the benefit of additional workers. llama.cpp’s generation troubleshooting guidance gives a deliberately conservative method: “If in doubt, start with 1 and double the amount until you hit a performance bottleneck, then scale the number down.”
- Run a repeatable generation test with one thread.
- Double the thread count and measure the same generation phase.
- Continue doubling while performance improves.
- When the result stops improving or becomes worse, return to the previous setting rather than keeping the largest number.
This is project-specific tuning advice, not a universal optimum for every runtime or processor. Leave enough CPU capacity for the operating system and any concurrent application. For a server, test the thread count at the concurrency level you actually need; a setting that wins for one interactive request may lose when several requests share the machine.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Separate prompt processing from token generation
Prompt processing, also called prefill, evaluates the existing input. Generation evaluates the model repeatedly to produce one new token at a time. They stress the hardware differently, so an optimization aimed at one phase may leave the other unchanged.
When BLAS can help
llama.cpp build documentation says BLAS may improve prompt processing when batch sizes are higher than 32. It explicitly does not improve generation performance. Therefore, BLAS is relevant when the delay is before the first token or when long prompts dominate total latency; it is not a promise of faster steady-state chat output.
Benchmark a prompt large enough to exercise the batch setting and report the prefill result separately. If generation speed is the complaint, keep investigating active offload, thread count, model fit, and memory pressure instead of judging BLAS by the wrong metric.
Control context length and memory pressure
Every additional context token requires memory for the model’s attention state. Allocate enough context for the application, but do not reserve a much larger window merely because the model advertises one.
Rank #4
- System Compatibility Note: This 2-slot card measures 271 x 112 x 39 mm and requires a single 12V-2x6-pin power connector. Please verify chassis and PSU compatibility before purchase.
- Dedicated Support: Please contact us directly through Amazon for any product questions or assistance you may require.
- Professional Intel Arc Pro B70 GPU: Built on the Intel Xe2-HPG architecture, it features 32 Xe cores and 256 XMX engines, designed to accelerate AI, rendering, and complex visualization workloads.
- Massive 32GB GDDR6 VRAM: Equipped with 32GB of high-speed GDDR6 memory on a 256-bit bus, running at 19 Gbps, which allows for handling large AI models and complex datasets locally.
- High-Performance Engine Clock: Delivers an engine clock of 2540 MHz, providing the compute power needed for demanding professional applications and AI inference.
NPU-specific OpenVINO behavior
The OpenVINO backend documentation warns that resolving an NPU context to the model’s full training context can cause an out-of-memory failure or degraded performance. It suggests setting an explicit context size, with -c 1024 shown as an example workaround. That value is documentation’s example, not a general recommendation: choose a size that covers your real prompts and output while fitting the device.
This warning is specific to the described NPU path. Do not assume that every runtime or every GPU handles context allocation in the same way. For any backend, watch both system memory and accelerator memory while testing, and treat swapping or allocation failures as a capacity problem rather than a thread-tuning problem.
Understand model size and quantization before buying hardware
Parameter count, quantization, context length, and runtime overhead determine whether a model fits and how much data must move during inference. A smaller or more aggressively quantized file can reduce memory demand, but quantization also changes model behavior and quality. The available evidence does not establish one universally best quantization or a cross-hardware buying ranking.
Use the model and quantization that meet your quality requirement, then verify that the weights and intended context fit in the memory available to the active backend. If the model only partly fits, the runtime may leave work on the CPU; that can be useful, but it changes the balance between prompt speed, generation speed, and memory traffic.
What the documented benchmark does—and does not—prove
llama.cpp troubleshooting documentation includes an illustrative 30B four-bit GGML model test on an NVIDIA A6000 with 48 GB of VRAM, seven physical CPU cores, and 32 GB of system RAM. Reported outcomes across different thread and GPU-offload configurations were 1.7, 5.5, 8.7, and 9.1 tokens per second. The documented -t 4 -ngl 2000000 run reported 9.1 tokens per second.
| Figure | What it describes | How to interpret it |
|---|---|---|
| 30B, four-bit GGML | Model configuration in the example | Not a universal model-speed category |
| 48 GB VRAM, seven physical CPU cores, 32 GB RAM | Hardware configuration | Results depend on this particular machine |
| 1.7, 5.5, 8.7, 9.1 tokens/s | Outcomes from different settings | Illustrates tuning sensitivity, not an expected result for another computer |
9.1 tokens/s at -t 4 -ngl 2000000 |
One specifically identified run | Do not reuse it as a buyer target |
The useful lesson is that settings changed the result on that setup. The numbers are not a cross-machine benchmark, a current market average, or a guarantee for another model, driver, or runtime version.
A repeatable tuning workflow
- Define the complaint. Decide whether you are optimizing time to first token, prompt-processing speed, generation tokens per second, total response time, or concurrent throughput.
- Capture a baseline. Record the model, quantization, context, prompt, runtime build, hardware, backend, threads, offload setting, and separate timing results.
- Verify backend activation. Read startup diagnostics for GPU-offloaded layers and memory use. If they are missing, repair the build or device selection first.
- Confirm memory fit. Check model weights, context allocation, system RAM, and accelerator memory during a run. Reduce unnecessary context only after confirming what the workload requires.
- Test CPU threads. In llama.cpp, start at one, double gradually, and step back after the bottleneck. Keep the best setting for the measured phase.
- Test prefill separately. If first-token latency dominates, evaluate batch behavior and BLAS. For llama.cpp, the documented BLAS benefit applies to prompt processing with batch sizes above 32, not generation.
- Repeat under the real workload. Re-test short and long prompts, expected output lengths, and the concurrency you actually plan to serve.
- Change hardware only after evidence. Consider an accelerator upgrade when the model fits a supported backend, offload is confirmed, and measurements show that additional compatible compute or memory addresses the bottleneck.
Why GPU inference can still be slow
The GPU is not doing the model work
A command may request many GPU layers while the executable lacks the matching backend, the device is hidden, or the runtime falls back to the CPU. Startup diagnostics are the quickest way to distinguish a real offload problem from a model that is simply demanding.
The model or context does not fit comfortably
Large weights plus a large context can exhaust accelerator memory or force part of the computation onto another device. On the documented OpenVINO NPU path, an implicit full training context can also cause allocation failure or degraded performance. Use an explicit, workload-appropriate context and observe memory while testing.
Recommended Free Tools
The measured phase is not helped by the setting
BLAS can improve llama.cpp prompt processing above the documented batch threshold but does not accelerate generation. Likewise, adding CPU threads to a generation bottleneck does not guarantee a gain. Always compare the setting with the metric it is supposed to affect.
The workload is different from the benchmark
Long prompts, short prompts, output length, quantization, backend, driver, and concurrency all change results. Reproduce your own request pattern instead of copying a tokens-per-second figure from a different model or machine.
How to compare runtimes, backends, or hardware
Use the same decision axes for every option:
| Axis | Question to answer |
|---|---|
| Compatibility | Does the runtime support this model, operating system, device, file format, and quantization? |
| Active acceleration | Do startup diagnostics prove that the intended backend is executing layers? |
| Memory fit | Do model weights and the required context fit without failure, swapping, or undesirable fallback? |
| Phase performance | Is the advantage in prompt processing, first-token latency, generation, or aggregate concurrent throughput? |
| Quality | Does the selected quantization meet the application’s quality requirement? |
| Reproducibility | Can you repeat the same workload and obtain comparable results after changing one variable? |
No single backend or hardware path wins on all six axes for every model. Compatibility and measured behavior on your workload matter more than a headline specification.
Symptom-based fixes
“The first response takes a long time, but streaming is fast.”
- Measure prompt-processing time and prompt token count.
- Check batch-related settings and, in llama.cpp, test BLAS for batches above 32.
- Reduce repeated conversation history or context only if the application can preserve the needed information.
- Do not claim success until time to first token improves; generation speed may remain unchanged.
“The first token is quick, but each following token is slow.”
- Verify GPU-offload diagnostics.
- Test llama.cpp thread counts from one upward, stopping after the bottleneck.
- Check whether model weights or context exceed practical accelerator memory.
- Compare generation tokens per second, not prompt-processing throughput.
“The GPU is busy, yet output is disappointing.”
- Confirm that the expected layers, rather than only a small portion, are offloaded.
- Check whether the model is split across CPU and GPU because of memory limits.
- Measure the exact model, quantization, context, and backend; a different benchmark is not a valid expectation.
“Performance collapses with long context on an Intel NPU.”
- Check the OpenVINO context behavior.
- Set an explicit context appropriate to the workload;
-c 1024is the documentation’s example, not a universal target. - Retest for both allocation success and latency.
“One chat is acceptable, but several users make it unusable.”
- Benchmark at the intended concurrency and record per-request latency plus aggregate throughput.
- Keep model, context, prompt, output length, and backend constant while changing concurrency.
- Check memory headroom and CPU contention before purchasing a faster component.
When an upgrade is justified
Hardware is a reasonable next step only after the software path is validated. An accelerator can help when the model and context fit a supported backend and measurements show that active offload or additional memory addresses the measured bottleneck. It will not fix a CPU-only build, an unavailable device, an unsupported quantization, or a benchmark that is actually dominated by prompt construction.
Free tools Windows power users keep installed
One-click scans. No signup required.
Choose an upgrade by the workload’s requirements: model memory, context memory, desired generation rate, prompt latency, and concurrent requests. Re-run the same benchmark on the candidate path and retain the setup details. Avoid treating a specification or an unrelated tokens-per-second claim as proof of the result you will obtain.
Quick Recap
Practical checklist
- Have you named the slow phase: prefill, first token, generation, total latency, or concurrency?
- Did you record model, quantization, context, runtime build, operating system, CPU, GPU, and memory?
- Did you repeat the same workload and change one variable at a time?
- Do startup logs prove that the intended accelerator backend is active?
- Have you tested CPU threads rather than assuming the largest value is fastest?
- Did you evaluate BLAS only for prompt processing where its documented benefit applies?
- Is context allocation appropriate for the backend, especially on the documented OpenVINO NPU path?
- Are your comparison numbers from the same model, workload, and hardware conditions?
- Have you verified software and memory fit before considering a hardware purchase?
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
