iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
There is no single speed switch for local inference on a Raspberry Pi 5. The practical fix is to identify whether the problem is thermal throttling, insufficient memory, runtime configuration, or unstable power, then test the same model and settings again.
Start by identifying the failure pattern
Different symptoms usually point to different constraints. Classify the failure before changing several settings at once; otherwise you can hide the original cause and make later comparisons meaningless.
| What you observe | Most likely constraint | First check |
|---|---|---|
| Generation is slow from the first token and remains steady | Model size, quantization, context length, CPU workload, or runtime overhead | Record the exact model, quantization, context, thread count, and runtime; then compare memory and benchmark results |
| Generation starts acceptably but becomes slower after several minutes | Heat and thermal throttling | Measure the SoC temperature during sustained generation and inspect the fan and airflow |
| The model fails to load, the Pi becomes unresponsive, or the system starts swapping | Insufficient RAM headroom, an oversized context, or another loaded model | Check free memory, unload other models, and lower the context setting |
| The server starts but a model is missing or cannot be selected | Incorrect model directory/layout, router mode, or model-loading policy | Use the runtime’s model and health endpoints and verify its serving configuration |
| The Pi reboots, disconnects, or reports power warnings under load | Power supply, cable, or high-draw USB peripherals | Check the supply against Raspberry Pi’s Pi 5 recommendation and temporarily remove nonessential USB devices |
| Short prompts work but long conversations become very slow or fail | Context and KV-cache memory growth | Reduce the context window and retest with a fixed prompt length |
Check for thermal throttling first when speed decays over time
A Pi 5 can produce acceptable output initially and then slow as the board reaches a sustained-load temperature. Cooling removes that thermal constraint; it cannot make an oversized model fit in RAM or overcome the normal limits of CPU inference.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsMeasure the SoC while the model is generating
- Run
vcgencmd measure_tempbefore starting inference. - Run it repeatedly while generating a long response. For a simple live view, use
watch -n 2 vcgencmd measure_temp. - Compare the temperature and token rate during the first minute with the rate after several minutes.
Raspberry Pi documentation notes that Linux temperature measurements can be inaccurate and recommends vcgencmd measure_temp for an instantaneous reading from the SoC. The command is a point-in-time sample, so one reading taken while idle does not diagnose throttling during generation.
#1 Best Overall
- Official RPi 5 Active Cooler -- This is Official RPi Active Cooler for the latest RPi 5 4GB/8GB Board
- Composition--The RPi 5 Active Cooler is composed of Temperature-controlled Blower Fan and Aluminium Heatsink and comes with Thermal Tapes to accelerate heat dissipation
- Input Voltage--5V DC (supplied via four-pin fan header on RPi 5)
- How to Install-- Connect the 4pin cable to the fan header on RPi 5, and fix the Active Cooler via spring-loaded push pins
- NOTE -- RPi 5 Board is NOT Included
Improve the cooling path
- Confirm that the fan is connected to the correct header, starts during load, and is not obstructed by the case.
- Improve air intake and exhaust around the board rather than placing it in a sealed enclosure.
- Use an official Pi 5 active-cooling option, such as the Active Cooler, when the workload is sustained.
- Use the Raspberry Pi 5 case fan when you need cooling integrated with the official case.
A broad Raspberry Pi thermal section describes progressive Arm throttling in the 80°C–85°C range and Arm/GPU throttling at 85°C. That passage is not presented as a Pi 5-specific operating threshold, so do not treat 85°C as a universal target or promise that every board will behave identically at that temperature. Use the measured temperature trend and the change in generation speed on your own system.
Retest without changing the model
After improving airflow or fitting a cooler, repeat the same prompt with the same runtime, context, thread count, and power supply. If the initial and sustained rates now remain close, heat was limiting the earlier run. If the rate is unchanged, continue with memory and runtime checks instead of adding more cooling.
Make the model, quantization, and context fit in memory
Model-file size is only part of the requirement. The operating system, inference runtime, model weights, temporary buffers, and the KV cache all need memory. A model can load successfully and still leave so little headroom that the Pi swaps, becomes unresponsive, or fails when the conversation grows.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Use the documented 8 GB example as a reference, not a specification
| Item | Reported value | How to interpret it |
|---|---|---|
| Board and RAM | 8 GB Raspberry Pi 5 | A single test system, not a requirement for every model |
| Model | Qwen3-8B Q4_K_M | The exact quantized build used in the report matters |
| Model file | 4.68 GiB | File size does not include all runtime and cache allocations |
| Memory while serving | About 5.2 GB | Observed in that setup; other runtimes, contexts, and services can use more or less |
| Context target | 4096 tokens | Described as sensible for that test system; a larger context needs more KV-cache memory |
The example demonstrates why an 8 GB board can serve an 8B quantized model while still having limited headroom. It does not establish a universal RAM requirement for Qwen3-8B, Q4_K_M, or any other model.
Rank #2
- This is Official Active Cooler for Raspberry Pi 5
- Combines an Aluminium Heatsink with a Temperature-Controlled Blower Fan to accelerate heat dissipation
- How to Install: Connect the 4pin cable to the fan header on RPi 5, and fix the Active Cooler via spring-loaded push pins
Check headroom before changing software
- With the model stopped, run
free -hand note available memory, not just total installed RAM. - Run the model and check memory again. If swap use rises or available memory approaches zero, stop other services and unload models that are not needed.
- Repeat the check after sending a long prompt. A growing context can consume additional KV-cache memory even when the model file itself has not changed.
- If the runtime exposes per-model memory information, record it with the model name and context value so that later tests remain comparable.
Reduce the largest memory consumers in the right order
- Lower context: In llama.cpp, the context option is commonly set with
-c; reducing it is specifically recommended when a load uses excessive memory. For the documented 8 GB example, 4096 tokens was a sensible starting point. Use the equivalent setting in other runtimes rather than passing llama.cpp flags to them. - Unload other models: A resident model can leave too little space for a second model or for a larger context.
- Choose a smaller model or lighter quantization: Lower memory use generally comes at the cost of capability or output quality. Keep the model identity and quantization in your benchmark notes.
- Reduce concurrency: Multiple parallel requests need additional working memory and can make an already constrained board appear frozen.
If lowering context makes the model load and the system responsive again, the previous failure was a memory-capacity problem, not a cooling problem. If memory is comfortable but generation remains slow, proceed to runtime and CPU measurements.
Verify the model-serving runtime and router configuration
A healthy inference process can still be unreachable or show no models because the server is pointing at the wrong directory, using the wrong mode, or waiting for an explicit load. Configuration is runtime-specific; commands for llama.cpp should not be copied into an Ollama setup.
llama.cpp router checks
The llama.cpp router guidance highlights these checks:
Free tools Windows power users keep installed
One-click scans. No signup required.
- Confirm the configured models directory and the directory/layout expected by the router. A file in an adjacent folder may be invisible even though it exists on the disk.
- Restart the router after adding or moving model files if the running process does not rescan them.
- Start router mode without a direct model-selection flag such as
--model,-m, or-hfwhen you intend the router to manage multiple models. - If the server was started with
--no-models-autoload, load the desired model through the router before attempting to select it. - Check the runtime’s health endpoint and models endpoint. A healthy process with an empty model list indicates a discovery or loading configuration problem, not necessarily a failed installation.
Use the endpoint names, port, and authentication settings from the version of llama.cpp you installed. Those details can change, so avoid assuming that a command copied from another release uses the same address.
Rank #3
- Compatible with Raspberry Pi 5 --- This Armor Lite V5 Aluminum Heatsink is only designed for Raspberry Pi 5 4GB/8GB.
- Support PWM Speed Control --- Different from ordinary fans, this cooling fan supports PWM speed regulation, which is perfectly compatible with Raspberry Pi OS.
- Good Heat Dissipation Effect --- With 3510 ultra-quiet cooling fan and thermal pads, it can lower the temperature of Raspberry Pi Board quickly.
- Lightweight and Easy to Install --- With screwdriver and 2pcs screws, it's easy to fix the heatsinks with Raspberry Pi Board.
- Package Includes: 1 x Armor lite V5 for Raspberry Pi 5, 1 x Screw driver, 2 x Screws, 4 x Thermal Pads, 1 x User Manual;
Ollama checks
Raspberry Pi Foundation project material documents a local LLM setup using Ollama. In that path, use Ollama’s own model-list, service-status, and log mechanisms to confirm that the model was pulled, the service is running, and the client is addressing the correct host. Do not apply llama.cpp router flags or directory conventions to Ollama.
Choose a runtime by the control you need
| Decision axis | Ollama | llama.cpp |
|---|---|---|
| Setup simplicity | Project material presents it as a straightforward local-LLM setup | Requires more direct management of model files and server options |
| Model compatibility | Depends on the models and formats supported by the installed Ollama release | Depends on the llama.cpp build and compatible model format |
| Context and serving control | Provides runtime-specific settings, but the exact controls depend on the release and API | Exposes direct controls for context, loading, routing, threads, and benchmarking |
| Diagnostics | Use its service status, model list, and logs | Use router health/models endpoints plus explicit model-loading state |
| Measured speed | No controlled head-to-head result is established here | No controlled head-to-head result is established here |
The available material does not prove that one runtime is categorically faster on every Pi 5. Compare them only with the same model file, quantization, context, thread count, parallel-request setting, cooling, power supply, and operating-system state.
Check power when the Pi reboots or reports warnings
Unexpected reboots, USB disconnects, or power warnings under inference are a different failure class from a stable but slow CPU workload. Raspberry Pi recommends its 27 W USB-C power supply for Raspberry Pi 5 and documents a 5 A recommended current capacity.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →| Power arrangement | Documented downstream USB allowance | Practical implication |
|---|---|---|
| 5 A supply | Up to 1.6 A for downstream USB peripherals | More margin for attached storage, cooling accessories, and other USB loads |
| 3 A supply | 600 mA for downstream USB peripherals | High-draw peripherals can consume the available margin sooner |
Power troubleshooting sequence
- Use a known-good 27 W Pi 5 supply and a suitable USB-C cable.
- Disconnect nonessential USB devices and retry the same inference request.
- Check whether the reboot or warning occurs only when a model is loading, when generation is sustained, or when a peripheral is active.
- Reconnect devices one at a time if the fault disappears, recording which combination brings it back.
A stronger supply is not a general speed upgrade. If the Pi is stable, has no power warnings, and only generates slowly, investigate model size, context, thermals, and runtime settings instead.
Rank #4
- KEEP YOUR RASPBERRY PI 5 COOL UNDER HEAVY LOADS - Designed for Raspberry Pi 5 4GB and 8GB models, the iUniker ICE PEAK Active Cooler helps move heat away during demanding workloads such as Home Assistant, Pi-hole, media servers, coding, retro gaming, and light AI projects.
- QUIET PWM FAN WITH ACTIVE AIRFLOW - The 3510 PWM fan adjusts speed based on Raspberry Pi 5 CPU temperature through Raspberry Pi OS. It runs quietly during light tasks and increases airflow when extra cooling is needed.
- ALUMINUM HEATSINK FOR HEAT DISSIPATION - The black aluminum heatsink helps spread heat across the finned surface, while the fan moves airflow through the heatsink to support stable cooling performance during extended use.
- LOW-PROFILE DESIGN FOR EXPANSION PROJECTS - The compact Pi 5 heatsink fan design helps leave more room for many Raspberry Pi 5 HATs, NVMe HATs, GPIO expansion boards, and stacked DIY builds. Please check clearance before use.
- EASY INSTALLATION WITH DIRECT FAN HEADER CONNECTION - We have pre-applied thermal grease on the heatsink. You only need to peel off the release paper and secure two screws to enjoy convenient cooling.
Set realistic speed expectations with a controlled benchmark
There is no single “Pi 5 tokens per second” figure. Generated-token speed changes with the model architecture and quantization, prompt length, context, runtime build, CPU frequency, thread count, parallel slots, cooling, and background services. Measure the exact combination you intend to use.
Documented example results
| Test configuration | Prompt processing | Generated output |
|---|---|---|
| 8 GB Pi 5; four Cortex-A76 cores; active cooling; llama.cpp CPU backend; Qwen3-8B Q4_K_M; four threads; one parallel slot; web UI runs at the reported 2.6–2.8 GHz profiles | 10.56–11.18 tokens/s | 2.13–2.15 tokens/s |
| Same model and board class; reported 3.0 GHz llama-bench runs | 11.45–11.50 tokens/s | 2.30–2.45 tokens/s |
These are sourced measurements from one configuration, not a typical-result guarantee. They should not be generalized to another model, context size, runtime build, cooling arrangement, or Pi variant. Prompt processing and generated output are different measurements: a high prompt rate does not mean that generated replies will arrive at the same rate.
Build a useful local benchmark
- Record the Pi 5 RAM capacity, operating-system image, runtime and version, model filename, quantization, context value, thread count, parallel slots, cooling, and power supply.
- Use a fixed prompt and fixed output limit. Keep the prompt short enough that context growth does not change between runs.
- Warm up the runtime once, then measure several runs rather than relying on the first request.
- Record prompt-processing and generated-token rates separately when the runtime reports both.
- Measure temperature during the run and note whether the speed decays over time.
- Change one variable at a time: context, threads, quantization, model, or runtime. A result obtained after changing several variables cannot identify which change helped.
If the controlled result is close to the documented example and the system is cool, stable, and responsive, the remaining limit is likely CPU-bound inference rather than a fault. A smaller model or lower-precision quantization may improve responsiveness, but it changes the quality and capability trade-off.
Common fixes that do not address the real cause
- Adding cooling to an out-of-memory model: A fan can reduce throttling but cannot provide the RAM required by a larger model or context.
- Buying a larger storage device for CPU-bound generation: An SSD can provide useful space and may improve file-management convenience, but the available evidence does not support it as a fix for slow CPU token generation.
- Increasing the power supply to cure stable slowness: The recommended supply matters when warnings, resets, or peripheral-current limits are involved; it does not increase generation speed on an otherwise stable board.
- Copying flags between runtimes:
-c, router options, model directories, and autoload policies are not interchangeable between llama.cpp and Ollama. - Judging performance from one short response: A brief request may not expose thermal throttling, context growth, or swapping. Include a sustained test.
- Comparing unlike benchmarks: Different quantizations, thread counts, frequencies, context windows, and web interfaces can overwhelm the difference you are trying to measure.
When the Pi is working as designed
After cooling has been checked, memory has adequate headroom, the router can see and load the model, and power is stable, a low but steady generation rate is not automatically a failure. The Pi 5 is running local inference on its CPU, so model size and quantization impose a real throughput limit. Keep the configuration that meets your response-time and quality needs, and describe it precisely whenever you compare results with another setup.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
