A local AI model can feel slow for three different reasons: it takes a long time to produce the first token, it processes a large prompt slowly, or it generates later tokens slowly. Identify which delay you have, then check the runtime’s CPU/GPU placement and memory use before changing settings or buying hardware. The right fix depends on the model, quantization, context length, runtime, hardware, and workload.
Why is my local AI model so slow?
First, separate the delay into the part you actually notice. A long pause before any response, a delay while the model reads your prompt, and slow word-by-word output have different likely causes. Record the same model and a short representative prompt before and after each change; otherwise, a different workload can make a setting look faster or slower than it is.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
MINISFORUM MS-02 Ultra Workstation Mini PC, Intel Core Ultra 9 285HX (24C/24T, up to 5.5GHz), PCIe... | $1,659.00 | Buy on Amazon |
| 2 |
|
GMKtec EVO-X2 AI Mini PC Ryzen Al Max+ 395 Superchip 128GB LPDDR5X 2TB SSD | $3,649.99 | Buy on Amazon |
- Slow time to first token: The model may be loading, processing a large prompt, or waiting on constrained memory or device placement.
- Slow prompt processing: Long prompts and long context settings can increase the work required before generation begins.
- Slow token generation: The model may be running partly on the CPU, have insufficient GPU memory, or be affected by CPU thread settings.
Do not treat tokens-per-second figures as directly comparable unless the model, quantization, context, runtime, hardware, and input/output lengths match.
Check what the runtime is doing before changing settings
A detected GPU does not prove that the whole model is using it. Depending on available VRAM, system memory, model size, and runtime configuration, inference can run on the CPU, GPU, or a split across both. While a request is active, inspect the runtime’s own status or diagnostic output for device placement and memory use.
Recommended Free Tools
#1 Best Overall
- High-Performance AI Processor：The MS-02 Ultra features an Intel Core Ultra 9 285HX (24C/24T, up to 5.5 GHz, 13 TOPS NPU), delivering fast and efficient performance for AI inference, algorithm development, and media workloads. A PCIe x16 expansion slot supports desktop-class GPU upgrades for advanced model training and accelerated computing tasks. It's ideal for creators, engineers, and teams handling intensive parallel workloads.
- 4 × M.2 PCIe 4.0 + 4 × DDR5 SODIMM slots:Four DDR5 SODIMM slots support up to 256 GB of memory, while ECC helps maintain data integrity in mission-critical environments. Four PCIe 4.0 M.2 slots support up to 24 TB of storage, supporting RAID 0/1/5/10, combining high-speed performance with data protection. It allows for the creation of independent scratch disks, media libraries, and project drives, providing high-throughput for production workflows.
- PCIe & USB 4.0 v2: Up to three PCIe slots can be equipped, including a dual-slot x16 GPU. The main slot supports PCIe 5.0, meeting the needs of high-bandwidth creative and computing workloads. USB 4.0 v2 (80Gbps) supports high-bandwidth external storage and displays.
- Ultra-fast Networking: Wi-Fi 7 further enhances wireless performance with next-generation speeds and low-latency stability. Intelligent bandwidth switching optimizes throughput in different network environments, ensuring optimal performance for enterprise or local networks. Dual 25GbE ports (providing up to approximately 3.125 GB/s bandwidth, about 25 times faster than traditional 1GbE), enabling seamless large-scale file transfers and parallel computing. 10GbE and 2.5GbE ports, with support for Intel vPro technology, ensure enterprise-grade remote management and deployment flexibility.
- Server-grade thermal architecture: Utilizing a dedicated CPU/GPU airflow design, equipped with a 6-pipe dual-fan cooler, it maintains stable performance even under sustained loads, delivering up to 140W Turbo power while maintaining a 100W TDP, and operating with noise levels as low as 36 dB. An integrated 350W power supply ensures stable and reliable output for demanding computing tasks and fully loaded extended configurations.
- Ollama’s FAQ describes model placement and memory considerations.
- For llama.cpp, consult its token generation performance tips for GPU offload diagnostics and thread guidance.
If the GPU is not being used as expected, check whether the runtime build and backend support your device and configuration. Changing context length or CPU threads will not fix a missing or unsupported GPU backend.
Make sure the model and context fit in memory
Model weights are only part of the memory budget. Runtime state and context also consume memory, and other GPU workloads can reduce what is available. If the model or its working state does not fit in VRAM, some work may fall back to the CPU or otherwise become constrained.
Try these changes one at a time if diagnostics indicate a memory or placement problem:
- Use a smaller model.
- Try a supported quantized version. Quantization reduces memory requirements, but it trades off quality and is not established as a universal speed improvement; check results on your own tasks.
- Reduce context length to what the job needs. In Ollama, context length can be configured; see its FAQ for the runtime’s guidance.
Test with a prompt representative of your actual use. Memory needs vary by model, context, runtime, and other active workloads, so a model’s advertised size alone does not establish whether it will fit.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Fix slow prompt processing and time to first token
Large prompts require more prompt processing before the model can begin answering. A longer context setting can also raise memory use, even when the current prompt is short. If the pause before the first token is the main problem, compare a short prompt with your usual prompt using the same model and settings. If the short prompt starts noticeably sooner, prompt size or context is a useful place to investigate.
- Keep the model, runtime, and other settings fixed.
- Run a short representative prompt and note the time to the first token.
- Run your normal prompt and compare the delay.
- If the longer prompt is much slower, trim irrelevant input or reduce the configured context to the minimum suitable for the task, then repeat the comparison.
Do not reduce context below what your task needs: the trade-off is that the model may no longer have room for all of the conversation or source material it must use.
Tune CPU threads if token generation is slow
If later tokens arrive slowly in llama.cpp, thread count can be worth testing even when GPU acceleration is involved. Its performance guidance recommends trying one thread as a diagnostic. If that improves token generation, the CPU may be oversubscribed; the documented next step is to set the thread count to the number of physical CPU cores rather than leaving an excessive value.
Rank #2
- EVOLUTION RYZEN AI MAX+ 395 MINI PC - GMKtec EVO-X2 is the next evolution in AI mini PC Ryzen Strix Halo series. Thanks to AMD Simultaneous Multithreading (SMT) the core-count is effectively doubled, to 32 threads. Ryzen AI Max+ 395 has 64 MB of L3 cache and can boost up to 5.1 GHz, depending on the workload. The Ryzen AI Max+ 395 is currently rated as the "most powerful x86 APU" on the market for AI computing.
- AI NPU with XDNA 2 ARCHITECTURE - Powered by 16 “Zen 5” CPU cores, 50+ peak AI TOPS XDNA 2 NPU and a truly massive integrated GPU driven by 40 AMD RDNA 3.5 CUs, the Ryzen AI MAX+ 395 is a transformative upgrade and delivers a significant performance boost over the competition. The Ryzen AI Max+ 395 excels in consumer AI workloads like the llama.cpp-powered application: LM Studio. Shaping up to be the must-have app for client LLM workloads, LM Studio allows users to locally run the latest language model without any technical knowledge required and unleash their creativity and productivity.
- AMD RADEON 8090S iGPU GAMING PC - The AMD Radeon RX 8060S offers all 40 CUs with up to 2.9 GHz graphics clock and uses the new RDNA 3.5 architecture. The powerful iGPU is positioned between an RTX 4060 and 4070 laptop GPU and therefore enables gaming in FHD at maximum details in most demanding games. The 8060S can also utilize the full 128GB pool, which is perfect for running LLMs such as Deepseek 70B Q8, which runs comfortably on this machine.
- EIGHT CHANNEL LPDDR5X - LPDDR5X is a new ground breaking memory small form factor installed on-board. With blazing speeds up to to 8000MT/s, it runs 1.5x faster than the DDR5 SODIMMs; 90% better performance over DDR5 SODIMMs in video conferencing and photo editing; 30% better performance in productivity apps; 12% better performance in digital content workloads.
- QUAD SCREEN 8K DISPLAY SUPPORT - EVO-X2 AI Mini PC support 4-screen 4K/8K output via HDMI 2.1 (8K@60Hz), DisplayPort 1.4 (4K@60Hz), and dual USB 4 40Gbps Transfer speed (supporting PD3.0/DP1.4/DATA). Ideal for gaming, video editing, and multitasking, it provides expansive and crisp multi-display support.
- Record the current thread setting and run the same prompt and model.
- Test with one thread and compare token generation speed.
- If one thread is faster, set the thread count to the number of physical CPU cores and test again.
This is a llama.cpp diagnostic path, not a universal thread setting for every runtime or workload. See the llama.cpp performance tips.
When batching and serving optimizations help
Batching and in-flight scheduling are designed to improve accelerator utilization and throughput when a system handles multiple requests. They do not necessarily reduce the latency a person notices while using a model alone. NVIDIA describes in-flight batching, KV caching, quantization, and speculative decoding for its serving configurations; those techniques are runtime- and workload-specific, not generic desktop settings.
For example, NVIDIA reported speculative-decoding throughput speedups of 3.55x, 3.16x, and 2.63x on a single H200 for Llama 3.3 70B using Llama 3.2 1B, Llama 3.2 3B, and Llama 3.1 8B draft models, respectively. These are vendor-reported results for specialized GPU serving, not expected improvements for a consumer PC. Details are in NVIDIA’s speculative decoding article.
When should you upgrade hardware?
Consider a GPU upgrade only after runtime diagnostics show that GPU memory or acceleration is a real constraint. The GPU needs enough usable VRAM for the model weights, runtime state, and intended context, with room for other GPU workloads. There is no one GPU recommendation that fits every model and use case. NVIDIA’s local AI guidance discusses VRAM planning and quantization.
More system RAM may matter for CPU inference or a system-memory constraint, but adding ordinary RAM should not be expected to speed up a workload that is already GPU-bound. A purchase decision should follow the diagnosed bottleneck, not the mere presence of slow output.
How to compare speed claims fairly
When evaluating a proposed setting, model, or hardware change, keep the test conditions visible. Useful comparisons include:
- Model and quantization
- Usable VRAM and whether the weights and runtime state fit
- Context length and prompt size
- Prompt-processing delay versus token generation speed
- Runtime version, backend, and GPU support
- Output quality for your task
- Cost, if comparing hardware
A vendor example illustrates why conditions matter: NVIDIA reported approximately 150 tokens per second on an RTX 4090 running Llama 3 8B with 100 input tokens and 100 output tokens. That is a vendor-reported result for those conditions, not a typical-speed promise or a fair comparison with a different model or setup. See NVIDIA’s llama.cpp on NVIDIA RTX Systems article.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
