iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Start with the 8GB Raspberry Pi 5 if you want to experiment with local language models. Raspberry Pi’s current guidance and Arm’s learning material both point to 8GB as the practical starting point; a 4GB board is better limited to small, quantized models with a short context window. Choose the model by its quantization, context length and runtime overhead—not just the parameter count or download size. For CPU inference, install 64-bit Raspberry Pi OS and use Ollama or llama.cpp. For accelerated local language or vision-language inference, use the AI HAT+ 2, which has 8GB of onboard memory and supports approximately six-billion-parameter LLMs and VLMs. The older AI HAT+ is for vision workloads and does not run LLMs.
How much RAM does a Raspberry Pi 5 need for an AI model?
An 8GB Raspberry Pi 5 is the preferred configuration for local LLM experimentation. Arm’s learning path states, “The 8GB RAM Raspberry Pi 5 model is preferred for exploring an LLM.” A 4GB board can run useful local models, but you need to stay with roughly 1B–3B-parameter, quantized models, keep the context window modest and leave room for Raspberry Pi OS and the inference program.
These are planning ranges rather than hard compatibility limits. Two models with the same parameter count can have different memory requirements because their quantization, vocabulary, context length and runtime buffers differ. A model file that fits on the storage card is not proof that it will fit in RAM while generating text.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitches| Pi 5 memory | Practical starting range | What to expect |
|---|---|---|
| 4GB | About 1B–3B parameters, quantized | Shorter contexts, fewer simultaneous services and less headroom for the operating system. Reduce context or unload other models if loading fails. |
| 8GB | About 3B–7B parameters for experimentation | More room for context and runtime buffers, while still requiring a suitable quantization and conservative settings. |
| AI HAT+ 2 | Approximately six-billion-parameter LLM/VLM support with 8GB onboard memory | Uses the Hailo accelerator path for supported models instead of relying entirely on the Pi CPU. |
The six-billion-parameter figure for AI HAT+ 2 is Raspberry Pi’s approximate support description, not a promise that every six-billion-parameter model, context length or quantization will fit. Check the currently supported model list before downloading.
#1 Best Overall
- HIGH PERFORMANCE: Features 26 TOPS (Trillion Operations Per Second) AI acceleration capability through the Hailo AI Accelerator for advanced machine learning applications
- COMPATIBILITY: Specifically designed for the Raspberry Pi 5, connecting via PCIe interface for optimal data transfer and processing speeds
- COMPACT DESIGN: Measures 65mm x 56.5mm, offering a space-efficient solution while maintaining full functionality as an AI acceleration add-on board
- TEMPERATURE RANGE: Operates reliably in temperatures from 0°C to +50°C (32°F to 122°F), ensuring stable performance in various environments
- SEAMLESS INTEGRATION: Functions as a HAT (Hardware Attached on Top) add-on board, providing plug-and-play compatibility with Raspberry Pi ecosystem
What uses the Pi’s memory when an AI model runs?
The parameter weights are only one part of the working set. The model file occupies storage, while the runtime also allocates buffers and a key-value cache for the context window. A longer prompt or conversation therefore increases RAM use even when the model itself does not change.
- Weights: the quantized tensors loaded from the model file.
- Context cache: memory used to retain your prompt and previous turns; larger context settings consume more.
- Runtime buffers: temporary memory used by the inference engine and, where applicable, accelerator transfers.
- Operating-system headroom: Raspberry Pi OS, the model server, a desktop session and other services must continue running.
When a loader reports that there is not enough memory, the first corrective action is usually to lower the context setting or unload another model. llama.cpp documents both approaches. Do not infer a safe RAM requirement by converting a model’s storage size directly into gigabytes of RAM.
Which models are sensible first choices on a Raspberry Pi 5?
For a first CPU-only experiment, choose a small quantized general-chat model. The Raspberry Pi Foundation’s starter guidance specifically recommends gemma:2b, phi or tinyllama. Select a coder-tuned model when programming quality matters, and select a Hailo-supported vision-language model (VLM) for camera or document tasks.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
| Use case | Starting choice | Memory strategy |
|---|---|---|
| General assistant or local chat | gemma:2b, phi or tinyllama |
Use a quantized build and begin with a conservative context on either Pi memory tier. |
| Programming help | A coder-tuned model in a size your RAM can hold | Prefer a smaller quantized model over an unquantized larger one; test with the context length you actually need. |
| Camera, image or document questions | A VLM listed by the current Hailo model service | Use AI HAT+ 2 and its supported Hailo package; do not assume a CPU LLM package supports image input. |
Raspberry Pi’s project guidance lists model storage from about 0.6GB for a 300-million-parameter model to 28GB for a 70-billion-parameter model. Those are storage figures, not guarantees of runtime RAM fit. The same guidance calls for at least a 32GB microSD card. A 70B download can therefore fit on a large drive while remaining impractical to load into a Pi 5’s available memory.
How should I compare quantization, context and storage before downloading?
Read the model’s quantization and context information before you start a large download. Ollama documents quantization levels including q4_K_M; llama.cpp uses GGUF files and exposes context controls. A lower-bit quantization generally reduces memory use at the cost of some quality, so compare the available variants rather than choosing the largest file automatically.
- Confirm the parameter tier. On 4GB, begin around 1B–3B parameters; on 8GB, investigate larger 3B–7B experiments only after allowing headroom.
- Record the quantization. Note whether the download is a quantized variant such as
q4_K_M, not merely the model family name. - Check the context window. Plan for the number of tokens your application will keep, because a high maximum context can consume substantial cache memory.
- Check the storage footprint. Keep at least the Foundation’s 32GB microSD baseline, and leave free space for the operating system, multiple model variants and updates.
- Choose the execution path. CPU inference uses Ollama or llama.cpp; supported accelerated models use AI HAT+ 2 and Hailo.
For reliability, keep model downloads on an SSD or a high-endurance microSD card when possible. Storage speed affects installation and model loading, but it does not increase the RAM available to the runtime.
What software and hardware do I need before installing?
Use a 64-bit installation of Raspberry Pi OS. Both the CPU and Hailo procedures depend on a 64-bit userland and current packages. Have network access for the initial download, a 32GB-or-larger microSD card for the base project, and enough free storage for the selected model.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minutePC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11- CPU route: Raspberry Pi 5, 64-bit Raspberry Pi OS, Ollama or llama.cpp, and a quantized model.
- Accelerated route: Raspberry Pi 5, AI HAT+ 2, 64-bit Raspberry Pi OS, the matching Hailo dependencies and the Hailo GenAI Model Zoo package.
- Thermal and power planning: sustained inference keeps the system busy; use a stable power supply and adequate cooling so the board can maintain its operating speed.
AI HAT+ and AI HAT+ 2 are not interchangeable for this job. Raspberry Pi describes AI HAT+ 2 as additionally allowing generative-AI models, while AI HAT+ is intended for vision workloads and does not support LLMs.
How do I install a model with Ollama on the Pi 5 CPU?
Ollama is the simplest CPU-oriented path for trying the Foundation’s starter models. Install it on 64-bit Raspberry Pi OS, download one model, and run it locally. The installation script is available at https://ollama.com/install.sh.
- Update Raspberry Pi OS and verify that you are running the 64-bit edition.
- Install Ollama with the documented command:
curl -fsSL https://ollama.com/install.sh | sh- Start a small model, for example:
ollama run gemma:2b- Send a short prompt first. Confirm that the model loads and responds before increasing the context or trying a larger model.
- If the model does not load, stop other model processes, choose a smaller quantized variant or reduce the context setting exposed by your client.
Ollama manages its model files, so monitor free space before pulling several variants. The 32GB card requirement is a project baseline, not a guarantee that every collection of models will fit. An SSD is preferable for repeated downloads and model swaps.
How do I run GGUF models with llama.cpp?
llama.cpp gives you direct control over GGUF files, context and optional offload. It is useful when you need to test several quantizations or run a local HTTP service with explicit limits.
Recommended Free Tools
- Install or build a current llama.cpp release for the 64-bit Pi environment.
- Place the chosen quantized GGUF files in a dedicated models directory, preferably on an SSD or high-endurance card.
- Start the router without forcing one model to load at startup. A conservative pattern is:
llama-server --models-dir /path/to/models --no-models-autoload --jinja --host 127.0.0.1 -c 32768 -ngl 999The exact executable name can vary by build; use the equivalent server command supplied by your llama.cpp package. The options in this pattern keep the service on the local machine, enable the model’s chat template where supported, set a 32,768-token context ceiling and request maximum layer offload where the build supports it.
- Load one GGUF model and test a short prompt.
- If loading fails, lower
-csubstantially or unload another model. A smaller context often succeeds without changing the model file. - Increase the context gradually while watching memory use and response stability.
-ngl 999 is a request, not a guarantee that every layer will be accelerated. On a CPU-only Pi, the runtime will use what the build and available hardware permit. Keep the server bound to 127.0.0.1 unless you have deliberately configured authentication and network access.
How do I install and use AI HAT+ 2 with Hailo?
Choose this route when you need accelerated local LLM or VLM inference and your desired model appears in the current Hailo list. AI HAT+ 2 provides 8GB of onboard memory and Raspberry Pi describes support for approximately six-billion-parameter LLMs and VLMs. The accessory’s supported model set can change, so use the list returned by the installed service instead of relying on an old model name.
Rank #2
- ⚡ PoE HAT for Raspberry Pi 5 CM5: PoE HAT F is a Power over Ethernet expansion board for Raspberry Pi 5 and CM5, supporting network connection and power input through one Ethernet cable.
- 🔌 802.3af/at PoE+ Support: This PoE+ HAT supports IEEE 802.3af/at network standard and works with compatible PoE power sourcing equipment for compact wired deployment projects.
- 🧊 Active Cooling Fan and Metal Heatsink: The PoE HAT with cooling fan includes a metal heatsink and high-speed active fan, helping improve heat dissipation and operating stability during long-term use.
- 🔋 5V and 12V Output Headers: Onboard 5V and 12V header outputs provide power options for external peripherals, with up to 25W total output under suitable PoE input and cooling conditions.
- 🧩 40-pin GPIO Stackable Header: Standard 40-pin GPIO stackable header fits Raspberry Pi 5 and CM5 expansion, allowing users to connect compatible HATs and custom project interfaces.
- Install 64-bit Raspberry Pi OS, update the system packages and update firmware.
- Install the Hailo dependencies that match your board and operating-system release.
- Install the Hailo GenAI Model Zoo package.
- Start the Hailo Ollama-compatible service,
hailo-ollama. - List models exposed by the local service:
curl http://localhost:8000/hailo/v1/list- Choose a model from that response and pull it through the local API. The official example includes
qwen2:1.5b, but treat that name as an example and follow the current list. - Send chat requests to the local API and verify that the response is using the Hailo service rather than a CPU-only fallback.
Use Hailo-supported VLMs for image or document prompts. Installing an arbitrary CPU GGUF does not make it compatible with the NPU; model format, compiler support and the installed GenAI package must match.
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →How can I tell whether a model fits before I commit storage and time?
Use a staged test instead of downloading the biggest model first. Begin with a small recommended model, measure whether it loads, then test the context length and workload you actually need.
- Check available RAM after boot and before starting the server.
- Pull one quantized model and run a short prompt.
- Open the longest realistic prompt or document and observe whether memory pressure causes an error or severe swapping.
- Only then try a larger parameter tier or a longer context.
- Keep one known-good small model so you can recover quickly after testing an oversized variant.
For a 4GB board, a 1B–3B quantized model with a modest context is the sensible baseline. For an 8GB board, 3B–7B experiments are more realistic, but the exact result still depends on quantization and context. Treat those ranges as planning guidance, not certification for a particular model release.
What should I do when loading fails or responses become unstable?
“Not enough memory” during load
Lower the context value first, then unload any other model or server process. With llama.cpp, reduce -c; with Ollama, stop unused models and select a smaller quantized variant. If the model still cannot load, move down a parameter tier.
The model fits but the Pi becomes unresponsive
The runtime may have left too little headroom for Raspberry Pi OS or other services. Use a smaller model, shorten the context, close the desktop and background applications, or move the workload to AI HAT+ 2 if the model is supported there.
The download fills the card
Delete unused model variants, place the model directory on an SSD and retain the 32GB baseline only as a minimum starting point. A larger card solves storage capacity, not runtime RAM.
Hailo service lists no suitable model
Confirm that the matching Hailo dependencies and GenAI Model Zoo package are installed, restart hailo-ollama, and query http://localhost:8000/hailo/v1/list again. If the model is absent from that response, it is not currently exposed by the installed service; do not substitute an arbitrary CPU model and expect NPU acceleration.
Output quality is disappointing
Quantization trades some quality for lower memory use. Try a higher-quality quantization that still fits, or use a smaller model tuned for your task rather than increasing parameter count blindly. Keep the context within the tested range so quality problems are not confused with memory pressure.
Which installation path should I choose?
| Your goal | Best starting path | Why |
|---|---|---|
| Learn local chat on a Pi 5 CPU | Ollama with gemma:2b, phi or tinyllama |
Fastest setup and model management for a small quantized model. |
| Control GGUF files and context explicitly | llama.cpp | Direct control over model directory, context, templates and layer-offload options. |
| Run supported LLMs or VLMs with acceleration | AI HAT+ 2 with Hailo | 8GB onboard memory and approximately six-billion-parameter support for models exposed by the current Hailo service. |
| Use the AI HAT+ accessory you already own | Vision workloads only | AI HAT+ does not support LLMs; use CPU inference or AI HAT+ 2 for generative models. |
What is the practical recommendation for a new build?
Buy or use the 8GB Pi 5, install 64-bit Raspberry Pi OS, and reserve at least a 32GB microSD card, preferably with an SSD for model storage. Start with a small quantized model such as gemma:2b, phi or tinyllama through Ollama. If you need larger contexts, explicit GGUF control or multiple model variants, move to llama.cpp and tune the context conservatively. Choose AI HAT+ 2 only when you need accelerated local LLM/VLM inference and your target model appears in the current Hailo list; the AI HAT+ vision board is not an LLM upgrade.
Frequently Asked Questions
Can a Raspberry Pi 5 run a 70B language model?
The Foundation’s table lists about 28GB of storage for a 70B model, but that is only the download footprint. A Pi 5 does not gain enough working memory simply because the file fits on a large card, so a 70B model is not a practical starting target for local Pi inference.
Is an 8GB Pi 5 faster than a 4GB model?
The extra memory mainly prevents loading and context failures; it does not by itself guarantee a proportional increase in token speed. CPU performance, quantization, context length and whether an accelerator is used determine observed throughput.
Can I increase the Pi’s RAM with a microSD card?
No. A microSD card or SSD adds model storage, while runtime weights, context cache and buffers still need RAM. More storage lets you keep more models; it does not make an oversized model load.
Do all Ollama models work with AI HAT+ 2?
No. Hailo acceleration depends on the model formats and versions supported by the installed Hailo GenAI package. Query the local model list and choose from the returned entries rather than assuming that every Ollama tag has an NPU implementation.
Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Repair Windows errors before they cause bigger problemsFix Now →What context length should I start with on a 4GB board?
Start with a modest context and increase it only after a short prompt works reliably. There is no universal safe token count because model architecture, quantization and background services change the memory headroom available on each installation.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
