iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes—you can run AI models entirely on a Raspberry Pi 5. The practical choices are CPU inference with Ollama or llama.cpp, or accelerated inference through Raspberry Pi’s AI HAT+ 2. CPU-only operation is the simplest and least expensive route, but it requires a small, quantized model, a deliberately chosen context window and effective cooling. AI HAT+ 2 adds a Hailo-10H neural-processing unit (NPU), 8 GB of onboard memory and an official workflow for local large-language and vision-language models up to about 6 billion parameters. All three approaches download the model to the Pi and can continue working without sending prompts to a cloud service after setup.
What can a Raspberry Pi 5 run locally?
A Raspberry Pi 5 can run a local conversational model, but it is not a desktop-class inference machine. The model’s parameter count, quantization level, context length and the memory available to the operating system determine whether a response is merely possible or comfortable to use. A compact quantized model is the sensible starting point for CPU-only use; larger models become more practical when an AI HAT+ 2 supplies dedicated acceleration and memory.
There are three useful routes:
- Ollama: the easiest starting point. It manages model files and provides a simple command-line interface; the Raspberry Pi Foundation also documents adding a browser-based interface.
- llama.cpp: a lower-level option when you need explicit control over GGUF models, loading, unloading and context size.
- AI HAT+ 2 with Hailo software: the official accelerated path for Raspberry Pi 5. It moves neural-network work to a Hailo-10H NPU rather than relying solely on the Pi CPU.
None of these options guarantees a particular tokens-per-second rate. Published TOPS and parameter limits describe hardware capability, not the application speed you will see with every model, prompt and context window.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWhat hardware and software do I need?
| Component | CPU-only Ollama or llama.cpp | Official Hailo LLM route |
|---|---|---|
| Board | Raspberry Pi 4 or 5 for the Raspberry Pi Foundation’s Ollama learning project; a Pi 5 is the focus here. | Raspberry Pi 5. |
| Operating system | A 64-bit Raspberry Pi OS installation is the practical baseline. | 64-bit Raspberry Pi OS Trixie is required by the current official workflow. |
| Accelerator | Not required; inference runs on the CPU. | AI HAT+ 2, using its Hailo-10H NPU. |
| Memory | Use a model whose quantized file and runtime fit the Pi’s available RAM with room for the operating system and context. | AI HAT+ 2 has 8 GB of onboard memory; Raspberry Pi documents LLM and VLM support up to about 6 billion parameters. |
| Storage | The Ollama learning project lists a microSD card of at least 32 GB. Larger models require more space. | Allow space for the operating system, Hailo software and downloaded models; the exact requirement depends on the models you keep. |
| Cooling | Use active cooling on the Pi 5 for sustained inference. | Use a Raspberry Pi 5 Active Cooler and install the heatsink supplied with AI HAT+ 2. |
| Internet | Required to install software and download models; not required for prompts after the model is stored locally. | Required for package and model setup; runtime requests can remain local. |
Use reliable power and storage as well. Inference can keep the processor busy for long periods, so an underpowered supply, a slow or failing card, or thermal throttling can look like a model problem.
#1 Best Overall
- HIGH PERFORMANCE: Features 26 TOPS (Trillion Operations Per Second) AI acceleration capability through the Hailo AI Accelerator for advanced machine learning applications
- COMPATIBILITY: Specifically designed for the Raspberry Pi 5, connecting via PCIe interface for optimal data transfer and processing speeds
- COMPACT DESIGN: Measures 65mm x 56.5mm, offering a space-efficient solution while maintaining full functionality as an AI acceleration add-on board
- TEMPERATURE RANGE: Operates reliably in temperatures from 0°C to +50°C (32°F to 122°F), ensuring stable performance in various environments
- SEAMLESS INTEGRATION: Functions as a HAT (Hardware Attached on Top) add-on board, providing plug-and-play compatibility with Raspberry Pi ecosystem
Which local-AI route should I choose?
Choose Ollama if you want the shortest path from a fresh Pi to a chat prompt. Choose llama.cpp if you want to decide exactly which GGUF files are available and when they are loaded. Choose AI HAT+ 2 if CPU-only performance or memory is inadequate and you are willing to use the Hailo-specific software stack.
| Decision factor | Ollama on CPU | llama.cpp on CPU | AI HAT+ 2 with Hailo |
|---|---|---|---|
| Setup difficulty | Lowest; the Foundation’s project provides a guided installation and web-interface path. | Moderate; you manage the server binary, model directory and runtime flags. | Highest; it requires Pi OS Trixie, firmware updates, kernel modules, Hailo packages and a separate model service. |
| Model control | Ollama handles model retrieval and runtime management. | Fine-grained control over GGUF files, loading and unloading. | Use models supported by the Hailo GenAI Model Zoo and the Hailo-Ollama service. |
| Acceleration | Pi CPU only. | Pi CPU unless you configure a compatible accelerator backend. | Hailo-10H NPU rated at 40 TOPS, with the Pi coordinating the application. |
| Practical model size | Small quantized models are the sensible starting point; no general official CPU throughput figure is stated. | Similar memory limits, but more control over context and loading. | Raspberry Pi documents LLM/VLM support up to about 6 billion parameters; actual fit and speed depend on the model and software. |
| Privacy | Prompts and generated text stay on the Pi when you use a locally stored model and do not add a cloud connector. | ||
| Cooling | Sustained workloads need active Pi 5 cooling; AI HAT+ 2 also needs its supplied heatsink. | ||
How do I run a model with Ollama?
Ollama is the beginner-friendly CPU route. The Raspberry Pi Foundation describes it as an open-source tool that runs large language models on the Pi itself rather than over the internet. Its learning project targets a Raspberry Pi 4 or 5, a microSD card of at least 32 GB and an internet connection for setup.
1. Prepare the Pi
- Install a current 64-bit Raspberry Pi OS release on the Pi 5 and complete the first-boot updates.
- Attach the Pi 5 Active Cooler before testing sustained generation. Keep several gigabytes of free storage beyond the operating system; a model download can consume more space than expected.
- Connect the Pi to the network for software and model downloads. You can disconnect it later if every required model and interface component is local.
2. Install Ollama and an interface
Follow the Raspberry Pi Foundation’s Ollama learning project to install the Ollama runtime and, if desired, its browser-based interface. The interface is optional: the command line is enough to validate that the runtime works before adding another service.
Free tools Windows power users keep installed
One-click scans. No signup required.
3. Download a model that fits
Use a small quantized model first. Quantization reduces the model’s memory footprint, while the context window adds working memory for the conversation. The largest model that can be downloaded is not necessarily the largest one the Pi can run reliably, so leave headroom for the operating system and the runtime.
ollama pull <model-name>Replace the code placeholder with a model identifier supported by your Ollama setup. Keep the model’s quantization and published memory guidance beside you; the available data does not establish one universal model name or a universal RAM threshold for every Pi 5 configuration.
4. Start a local chat
ollama run <model-name>Enter a short prompt and observe both response time and memory use. A first response can be slower while the model is loaded. Subsequent prompts are a better indication of steady-state behavior, but a long conversation can increase memory pressure as the context grows.
5. Add the web interface only after the command line works
A browser interface makes the Pi easier to use from another device, but it introduces another process and a network listener. Verify the command-line model first, then configure the interface according to the Foundation project’s instructions. Keep the interface bound to your local network unless you have deliberately added authentication and firewall rules.
How do I use llama.cpp for more control?
llama.cpp is useful when you want to manage GGUF models yourself instead of relying on a higher-level model catalog. Its documented router discovers models in a models directory and can load or unload them on demand. Explicitly setting a context size is important: the documentation warns that omitting it can require substantially more memory.
1. Arrange a models directory
Place compatible GGUF files in one directory and note the path. Keep only models you actually intend to use; every additional file consumes storage even when it is not loaded.
Rank #2
- This kit includes an AI HAT+, a metal case and an active cooler. It's compatible with Raspberry Pi 5.
- The Raspberry Pi AI HAT+ features a built-in neural network accelerator, turning your Raspberry Pi 5 into a high-performance, accessible, and power-efficient AI machine.The 13 TOPS variant capably runs neural networks for applications including object detection, semantic and instance segmentation, pose estimation, and more.
- The AI HAT+ communicates using Raspberry Pi 5’s PCIe Gen 3 interface. When the host Raspberry Pi 5 is running an up-to-date Raspberry Pi OS image, it automatically detects the on-board Hailo accelerator and makes the NPU available for AI computing tasks. The built-in rpicam-apps camera applications in Raspberry Pi OS natively support the AI module, automatically using the NPU to run compatible post-processing tasks.
- Conforms to Raspberry Pi HAT+ specification; Supplied with 16mm stacking header, spacers, and screws to enable fitting on Raspberry Pi 5 with Raspberry Pi Active Cooler in place.
- The metal case can protect the Raspberry Pi 5 board from damage, dust and scratches. It can access most ports, including usb-c power jack, micro HDMI ports, usb ports, Ethernet jack, sd card slot, power button and GPIO port.
2. Start the server with an explicit context
The following pattern uses the documented options. Set the context value conservatively for your available memory, then increase it only after a smaller test succeeds.
./llama-server --host 127.0.0.1 --port 8080 --models-dir ./models --no-models-autoload --jinja -ngl 999 -c 2048--host 127.0.0.1keeps the API on the Pi itself rather than exposing it to the network.--port 8080selects the local server port.--models-dir ./modelspoints the router at your GGUF files.--no-models-autoloadprevents every discovered model from being loaded automatically.--jinjaenables the model’s Jinja chat-template handling where supported.-ngl 999requests maximum layer offload when a usable accelerator backend exists; on a CPU-only Pi, it does not turn the CPU into an NPU.-c 2048sets an example context size. Treat it as a starting value, not a guarantee that every model will fit.
3. Load one model and test it
Use the server’s model-selection controls to load one GGUF file, send a short prompt and watch memory. If the process is killed, the system becomes unresponsive or generation stops during a long prompt, unload the model, reduce the context, or select a smaller quantization before trying again.
4. Keep the server local unless you secure it
The documented default binds to loopback, so requests originate on the Pi. To make it reachable from another computer, you would need deliberate network configuration and access controls; do not expose an unauthenticated inference endpoint directly to the internet.
How do I add AI HAT+ 2 acceleration?
AI HAT+ 2 is the current Raspberry Pi option for local LLM and VLM acceleration. Raspberry Pi specifies a Hailo-10H NPU rated at 40 TOPS and 8 GB of onboard memory, and documents support for models up to about 6 billion parameters. Those are hardware and software-support figures, not a promise that every supported model will deliver the same latency or fit every workload.
1. Confirm the prerequisites
- Raspberry Pi 5 running 64-bit Raspberry Pi OS Trixie.
- AI HAT+ 2 correctly installed on the Pi.
- Raspberry Pi 5 Active Cooler fitted.
- The additional AI HAT+ 2 heatsink installed on top of the HAT, as Raspberry Pi recommends.
- Network access for firmware, package and model installation.
2. Update the operating system and firmware
Bring Raspberry Pi OS and the Pi firmware up to date before installing the accelerator stack. Reboot if the updater requests it; mixing old firmware with a newly installed kernel module is a common source of detection failures.
3. Install the kernel and Hailo packages
sudo apt update
sudo apt full-upgrade -y
sudo apt install -y dkms hailo-h10-all
sudo rebootThe dkms package allows the driver component to integrate with the running kernel. Installing hailo-h10-all supplies the Hailo H10 software set described by Raspberry Pi’s workflow.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →4. Verify the accelerator
hailortcli fw-control identifyA successful identification confirms that the firmware-control utility can see the Hailo device. If it cannot, power down and reseat the HAT, check the cooling and connector installation, confirm that the Pi 5 firmware update completed, and rerun the command after rebooting.
5. Install the GenAI Model Zoo package
Install the Hailo GenAI Model Zoo package specified by Raspberry Pi’s AI software documentation. This package is the compatibility layer for the documented generative-AI workflow; a random GGUF file that works with llama.cpp is not automatically a Hailo model.
6. Start Hailo-Ollama and list available models
Start the hailo-ollama service using the launcher supplied with the installed Hailo package. Then query its local model-list endpoint:
Rank #3
- The Raspbery Pi AI HAT+ is an add-on board with a built-in Hailo AI accelerator designed for RPi 5. It provides an accessible, cost-effective, and power-efficient way to integrate high-performance AI. It's suited to everything from entry-level applications to more complex neural processing, with the ability to process multiple concurrent models and AI tasks. Explore applications including process control, security, home automation, and robotics.
- This AI HAT+ is available in 13 TOPS variants, built around the Hailo-8L neural network inference accelerators. The 13 TOPS variant capably runs neural networks for applications including object detection, semantic and instance segmentation, pose estimation, and more.
- The AI HAT+ communicates using Raspbery Pi 5's PCIe Gen 3 interface. It automatically detects the onboard Hailo accelerator and makes the NPU available for AI computing tasks. The built-in rpicam-apps camera applications in Raspbery Pi OS natively support the AI module, automatically using the NPU to run compatible post-processing tasks.
- Hailo-8L accelerator offering 13 TOPS inferencing performance respectively. Fully integrated into Raspbery Pi's camera software stack. Conforms to Raspbery Pi HAT+ specification.
- Comes with 16mm stacking header, spacers, and screws to enable fitting on Raspbery Pi 5 with Raspbery Pi Active Cooler in place.
http://localhost:8000/hailo/v1/listThe response shows which models the service can offer. Select a model from that list rather than assuming that a CPU-oriented Ollama model is compatible.
7. Pull a model and send a chat request
Use Hailo-Ollama’s model-pull operation for a listed model, wait for the download and compilation steps to complete, then send chat requests to:
/api/chatKeep the first prompt short while validating the installation. NPU acceleration moves neural-network work away from the Pi CPU, but the Pi still handles orchestration, input/output and parts of the application; cooling and adequate storage remain relevant.
How much RAM, storage and context do local models need?
There is no single RAM number that applies to every model. A model’s quantized weights, runtime buffers, tokenizer, conversation context and operating-system processes all compete for memory. The same parameter count can therefore behave differently at different quantization levels and context lengths.
Use quantization to stay within memory
Quantized models store weights with fewer bits than full-precision versions. They are usually the only practical choice for CPU-only inference on a Pi 5, but lower memory use can involve a quality or accuracy trade-off. Select a quantization explicitly and verify that the file size leaves working-memory headroom; do not equate file size with total runtime memory.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Treat context as a configurable budget
Every token retained from the prompt and conversation consumes additional memory. Start with a modest context in llama.cpp, avoid pasting entire documents into a first test, and increase the setting only when short prompts remain stable. In Ollama, keep early conversations short while you establish a baseline, because a long chat can become the limiting factor even when the model itself initially loads.
Measure instead of guessing
Before launching a large model, check available memory with the operating system’s standard tools and leave room for the desktop, web interface and background services you actually run. During a test, watch for the kernel terminating the process, swap activity, severe slowdown or an abrupt reboot. Those symptoms indicate that the selected model or context exceeds the practical memory budget.
Plan storage for more than one model
The Foundation’s Ollama project sets 32 GB as the minimum microSD capacity for its setup, not as a guarantee that every model will fit. Keep extra capacity for the OS, updates, caches and model variants. If you keep several quantizations, calculate the total of all model files before downloading them.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why is cooling essential on a Pi 5?
Local generation can hold the CPU under sustained load, unlike a brief compilation or command-line task. As temperature rises, the Pi can reduce clock speed to protect itself, making generation slower and less consistent. Fit the Raspberry Pi 5 Active Cooler before comparing models or judging performance.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Rank #4
- The Vilros Raspberry Pi 5 AI Kit Provides a full set of hardware needed to get up and running with your AI Projects.
- Kit Includes: Raspberry Pi 5 (Choose Capacity)--Raspberry Pi AI HAT+ (Choose TOPS Capacity)--Raspberry Pi 5 Active Cooler--Vilros Raspberry Pi 5 + Hat Compatible Case--128GB Micro SD Card Preloaded W/ Raspberry Pi OS (64bit)--Vilros 27W -5V/5A Raspberry Pi 5 Compatible USB-C Power Supply--Vilros Micro HDMI to Standard HDMI Cable (5ft)--Vilros Neoprene Parts Storage Case Bag With Pocket--Vilros Micro SD to USB Adapter
- Powerful Performance: Raspberry Pi 5 offers a 3× increase in CPU performance with a 2.4GHz quad-core Cortex-A76 processor. Enjoy smoother, faster computing for DIY projects, programming, or home automation. .
- Hailo-8 or Hailo-8L accelerator ( 26 TOPS or 13 TOPS Variants Available) -Fully integrated into Raspberry Pi’s camera software-Supplied with 16mm stacking header, spacers, and screws to enable fitting on Raspberry Pi 5 with the included Raspberry Pi Active Cooler in place
AI HAT+ 2 has its own thermal requirement: Raspberry Pi supplies an additional heatsink and recommends installing it on top of the HAT. The Pi cooler and HAT heatsink solve different heat paths, so omitting either one can undermine a build that otherwise has adequate software.
- Check that the fan spins and that the heatsink makes proper contact.
- Keep the board in a ventilated case; do not cover the fan intake or exhaust.
- Test a sustained prompt rather than relying on an idle temperature reading.
- If performance drops over time, inspect temperature and throttling before changing models.
What are the privacy and offline implications?
With Ollama, llama.cpp or Hailo-Ollama configured for a downloaded local model, prompt text and responses remain on the Pi. Internet access is needed for initial operating-system updates, package installation and model downloads, but it is not inherently required for subsequent local prompts.
“Local” does not automatically mean “isolated.” A browser interface, telemetry component, package updater or remote API connector can still make network requests. If offline operation matters, disconnect the network after setup and verify that the chosen interface still functions; keep a copy of the model and installation notes on local storage.
Binding llama.cpp to 127.0.0.1 limits its documented server to the Pi. If you deliberately make a service reachable from another device, protect it with network segmentation, authentication and firewall rules appropriate to your home or lab.
Recommended Free Tools
How do I troubleshoot a failed local-model setup?
The model will not load or the process is killed
- Choose a smaller quantized model.
- Lower the context size, especially in llama.cpp where an omitted context can consume substantially more memory.
- Stop an unused web interface or other memory-heavy process.
- Check available RAM and swap while loading, not only after the failure.
Generation becomes slower after several minutes
- Confirm that the Pi 5 Active Cooler is connected and unobstructed.
- Install the AI HAT+ 2 heatsink if using the NPU route.
- Check for thermal throttling and improve ventilation.
- Repeat the test with a shorter context to separate thermal issues from memory growth.
llama.cpp cannot find a model
- Verify that the file is a supported GGUF model.
- Confirm that the path passed to
--models-diris the directory containing the file. - Keep
--no-models-autoloadwhile troubleshooting so an unintended large model is not loaded. - Load one model at a time and check its name in the router’s model list.
The Hailo device is not detected
- Run
hailortcli fw-control identifyafter rebooting. - Recheck that the board is a Raspberry Pi 5 running 64-bit Raspberry Pi OS Trixie.
- Confirm that both
dkmsandhailo-h10-allwere installed successfully. - Power down, reseat the AI HAT+ 2, inspect the connector and heatsink installation, then boot again.
The Hailo model list is empty
Make sure the Hailo GenAI Model Zoo package is installed and that Hailo-Ollama is running. Query http://localhost:8000/hailo/v1/list locally before attempting a pull; an empty list usually means the service or model package is not ready rather than that a CPU GGUF file is missing.
Which setup is the best starting point?
For a first experiment, use Ollama on a cooled Pi 5 with one small quantized model. Move to llama.cpp when you need explicit control over GGUF files or context allocation. Add AI HAT+ 2 when you need the documented Hailo acceleration path for LLMs or VLMs and accept its extra hardware, operating-system and model-compatibility requirements. In every case, begin with a short prompt, measure memory and temperature, and scale the model or context only after the baseline is stable.
Frequently Asked Questions
Can a Raspberry Pi 5 run a local model without an internet connection?
Yes, after the operating system, runtime and model files have been downloaded. Disconnecting the network can expose hidden dependencies in a browser interface or updater, so test the exact offline arrangement you plan to use.
Can I use any Ollama or GGUF model with AI HAT+ 2?
No. The Hailo workflow uses models exposed by the Hailo GenAI Model Zoo and Hailo-Ollama service. Compatibility with CPU-oriented Ollama or llama.cpp files is not automatic.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errorsWhy does increasing the context window make the Pi unstable?
The context is additional working memory on top of the model weights and runtime buffers. A model that loads at a short context can run out of RAM, enter swap or be terminated when a much longer conversation is retained.
Can another computer use the model running on my Pi?
Only if you intentionally expose the service beyond loopback and secure it. The documented llama.cpp setup binds to 127.0.0.1, which keeps requests on the Pi until you change the network configuration.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
