iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
For a CPU-only Raspberry Pi 5, Gemma 4 E2B is the strongest currently measured general-purpose choice, while Gemma 3 270M is the fastest and lightest option in Raspberry Pi’s published tests. Choose Gemma 3 1B for multilingual generation, EmbeddingGemma 300M for embeddings and retrieval, and Gemma 3 270M for narrow classifiers or extractors. If you want an officially supported accelerator, use the Raspberry Pi AI HAT+ 2: it has a Hailo-10H processor, 40 TOPS INT4 performance and 8 GB of onboard memory, and Raspberry Pi documents local LLM support for models up to approximately 6 billion parameters.
There is no single best model for every Pi 5. Your decision depends on available RAM, whether you are willing to add the AI HAT+ 2, the task’s quality requirements, context length, power and cooling, and whether you need a conversational model or a narrow embedded component.
Quick recommendations
| What you need | Best starting point | Why | Main limitation |
|---|---|---|---|
| Lightweight CPU chat or text generation | Gemma 4 E2B with LiteRT-LM | Raspberry Pi measured 9 decode tokens/sec and 1,432 MB peak memory on an 8 GB Pi 5. | The result applies to the tested build and settings, not every runtime or Pi configuration. |
| Lowest memory use and highest published CPU decode speed | Gemma 3 270M | Raspberry Pi reports 22.58 decode tokens/sec and 680 MB peak memory with LiteRT-LM. | Its small size limits broad knowledge and complex reasoning; Raspberry Pi positions it for task-specific fine-tuning. |
| Embeddings, retrieval or semantic search | EmbeddingGemma 300M | Raspberry Pi describes it for embeddings, RAG, semantic search and classification. | It is not a general conversational model. |
| Multilingual generation | Gemma 3 1B | Raspberry Pi describes it as suitable for multilingual generative tasks. | It requires more memory and compute than the sub-billion-parameter models. |
| Coding on an accelerated Pi | Qwen2.5-Coder 1.5B on AI HAT+ 2 | It was one of the models demonstrated or listed at the AI HAT+ 2 launch. | Availability depends on the current Hailo model catalog and compatible package. |
| Supported local LLM acceleration | AI HAT+ 2 with a Hailo-compatible model | The add-on provides dedicated memory and LLM/VLM support. | The older AI HAT+ is vision-focused and does not support LLMs. |
What “best” means on a Raspberry Pi 5
A model that is excellent on a desktop can be a poor Pi 5 choice. The useful comparison is not parameter count alone; it is the combination of task quality, memory footprint, first-token delay, decode speed, supported format, runtime compatibility and sustained thermal behavior.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Decode speed matters most during an interactive reply
Prefill is the phase in which the runtime reads your prompt and existing context. Decode is the generation of new tokens. Decode tokens per second usually has the greatest effect on how quickly a chat response appears. A model can show a high prefill rate yet still feel slow if its decode rate is low.
#1 Best Overall
- All-in-One AI Learning Lab Powered by Raspberry Pi & Multi-LLMs. Turn Raspberry Pi (5 / 4B / 3B+ / 3B / Zero 2W) into a complete AI learning lab with support for multi-LLMs like ChatGPT, Gemini, Grok, DeepSeek, Qwen, Doubao, and Ollama. Includes Pan-Tilt HAT,10-axis (10DOF) module, camera, and high-quality components. Learn AI through guided video lessons created with educator Paul McWhorter. (Raspberry Pi not included)
- Build Fun Multi-Modal AI Projects with Voice, Vision & Sensors. Combine sensors, breadboard circuits, Multi-LLMs, voice recognition, and camera vision to create engaging multi-modal AI projects. Learn STT and TTS through hands-on programming, turning abstract AI concepts into interactive projects you can see, hear, and control—perfect for AI beginners
- AI Vision Tracking with YOLO, OpenCV, MediaPipe & Pan-Tilt HAT. Create intelligent vision projects using OpenCV and MediaPipe to detect and track objects, colors, and human movements. The Pan-Tilt HAT allows your projects to actively follow targets, helping learners understand how AI vision and motion work together in real systems
- Fusion HAT+ Power System with Voice AI Interaction. The Fusion HAT+ provides power, safe shutdown, and simplified hardware control via a unified Python library. With the Fusion HAT+ featuring a built-in speaker and microphone, easily build AI voice interaction projects by combining Multi-LLMs with sensors and electronic components
- Step-by-Step Learning with Video Lessons & Technical Support. Includes a structured, project-based curriculum with clear documentation, sample code, and video tutorials created with Paul McWhorter. Backed by responsive technical support and an active community, this kit helps beginners confidently progress from Python basics to AI and interactive projects
Memory determines whether a model is practical
Pi 5 system RAM must hold the operating system, runtime, model weights, KV cache and application overhead. Peak memory is therefore more useful than a model’s advertised parameter count when deciding whether a workload will fit comfortably. Context length also increases memory use, so a short benchmark prompt does not prove that a long document or extended conversation will run equally well.
Parameter count is only a rough filter
Raspberry Pi’s tutorial advises caution with models above 5 billion parameters on a standard Pi. That is a warning, not a promise that every model below 5 billion parameters will be fast, accurate or compatible. Quantization, architecture, context length and runtime can change the result substantially.
Best CPU-only models
Gemma 4 E2B: the best measured general-purpose starting point
For a general model on a Pi 5 without an accelerator, Gemma 4 E2B is the most useful current reference point because Raspberry Pi published measurements for it on an 8 GB board. The August 11, 2026 test used 1,024 prefill tokens, 256 decode tokens and four CPU threads.
Free tools Windows power users keep installed
One-click scans. No signup required.
| Runtime and build | Prefill | Decode | Peak memory |
|---|---|---|---|
| LiteRT-LM, QAT | 99 tokens/sec | 9 tokens/sec | 1,432 MB |
| llama.cpp, Q4_0 | 24 tokens/sec | 4 tokens/sec | 4,406 MB |
These figures are not interchangeable. The LiteRT-LM and llama.cpp rows use different quantization or build paths, so they show how strongly runtime and model packaging affect performance. They are measurements for the stated Pi 5 setup, not universal guarantees for every Gemma 4 E2B download, prompt length or operating-system image.
Gemma 4 E2B is a sensible choice when you want one model for ordinary questions, short explanations and constrained edge applications. Keep expectations realistic: small edge models do not contain the knowledge breadth of large cloud models, and they are better suited to bounded datasets and clearly defined tasks.
Gemma 3 270M: fastest and lightest for narrow tasks
Raspberry Pi’s appendix reports 22.58 decode tokens/sec and 680 MB peak memory for Gemma 3 270M with LiteRT-LM. That combination makes it attractive when responsiveness and RAM headroom matter more than broad generative ability.
Raspberry Pi specifically describes this model as suitable for task-specific fine-tuning, including sentiment analysis and entity extraction. Those are better fits than open-ended research or long-form reasoning. A small classifier that performs one job reliably can be more useful in an embedded product than a larger chat model that consumes more memory and produces inconsistent output.
EmbeddingGemma 300M: use it when generation is not the job
EmbeddingGemma 300M is intended for embeddings, retrieval-augmented generation, semantic search and classification. An embedding model converts text into vectors that an application can compare, rather than composing a conversational answer token by token.
Choose it for a local document index, similarity search, duplicate detection or a retrieval stage that feeds another model. Do not select it as your only model if the requirement is a chat interface; it solves a different part of the pipeline.
Gemma 3 1B: a middle ground for multilingual generation
Raspberry Pi describes Gemma 3 1B for multilingual generative tasks. It is the logical CPU candidate when language coverage is more important than the lowest possible memory use, but a larger model is not practical.
Because the published figures do not include a Gemma 3 1B speed or memory measurement, treat it as a task-based candidate rather than a performance promise. Test the languages, prompt format and context length you actually plan to deploy.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Rank #2
- Includes Raspberry Pi 5 with 2.4Ghz 64-bit quad-core CPU (8GB RAM)
- Includes 128GB Micro SD Card pre-loaded with 64-bit Raspberry Pi OS, USB MicroSD Card Reader
- CanaKit Turbine Black Case for the Raspberry Pi 5
- CanaKit Low Noise Bearing System Fan
- Mega Heat Sink - Black Anodized
How to interpret Raspberry Pi’s CPU benchmark
The published Gemma 4 E2B comparison answers a narrow question: how two specified software paths behaved on an 8 GB Pi 5 under a defined token workload. It does not establish a complete ranking of all small models.
- Board tier: the test used an 8 GB Raspberry Pi 5. Results may not transfer directly to a lower-memory board.
- Thread count: four CPU threads were used. Changing thread count can alter both speed and contention with other services.
- Prompt shape: 1,024 prefill tokens and 256 decode tokens do not represent every chat, document or coding workload.
- Runtime and quantization: LiteRT-LM with QAT and llama.cpp with Q4_0 have different memory and speed characteristics.
- Peak memory: 1,432 MB versus 4,406 MB is a major practical difference, especially when the Pi is also hosting a database, web server or camera pipeline.
Use the figures as a baseline for planning, then validate the exact model file and runtime on the board that will ship. Do not turn the benchmark into a population statistic or assume that a faster decode rate means better answers.
When the AI HAT+ 2 is the better route
The AI HAT+ 2 is Raspberry Pi’s documented Pi 5 add-on for local LLM work. It includes a Hailo-10H accelerator, 40 TOPS INT4 performance and 8 GB of onboard memory. Raspberry Pi documentation says models up to approximately 6 billion parameters can fit, subject to supported formats and the current software stack.
The separate, older AI HAT+ is vision-focused, uses Pi system RAM and does not support LLMs. Buying it for language-model inference is therefore a compatibility mistake.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteModels named for the AI HAT+ 2 launch
Raspberry Pi’s January 15, 2026 announcement listed these launch models:
| Model | Size stated in the announcement | Likely role |
|---|---|---|
| DeepSeek-R1-Distill | 1.5B | Reasoning-oriented experimentation |
| Llama 3.2 | 1B | General text generation |
| Qwen2.5-Coder | 1.5B | Coding assistance |
| Qwen2.5-Instruct | 1.5B | Instruction-following chat |
| Qwen2 | 1.5B | Questions and translation; used in Raspberry Pi’s demonstration |
This is a dated launch list, not a complete current catalog. Before deployment, check the current Hailo model list and package compatibility. Raspberry Pi also describes LoRA-based task-specific adaptation on this route, which can be useful when a compact model needs to follow a specialized vocabulary or workflow.
What acceleration does and does not guarantee
The HAT moves supported inference work onto dedicated hardware and supplies its own memory, reducing pressure on the Pi’s system RAM. It does not guarantee that every model will load, that every quantization will be supported, or that the output quality will match a large cloud model. Compatibility with the Hailo Gen-AI Model Zoo, the Hailo Ollama server and the installed software version is part of the decision.
CPU-only versus AI HAT+ 2
| Decision factor | CPU-only Pi 5 | AI HAT+ 2 |
|---|---|---|
| Extra hardware | None beyond the Pi and its normal cooling | Hailo-10H add-on with mounting hardware and optional heatsink |
| Memory location | Uses Pi system RAM | 8 GB onboard accelerator memory plus Pi system resources |
| Supported model path | Depends on the chosen CPU runtime and model build | Hailo Gen-AI Model Zoo and compatible Hailo Ollama packages |
| Best use | Learning, prototypes, narrow models and modest chat workloads | Supported local LLM/VLM inference and repeatable accelerator deployments |
| Quality comparison | No controlled CPU-versus-HAT model-quality comparison is established; test the task itself. | |
| Setup burden | Lower hardware complexity, but runtime choice remains important | Requires physical installation and the Hailo software stack |
| Upgrade flexibility | Broad experimentation if a CPU runtime supports the model | More dependent on the current Hailo catalog and compatible formats |
Choose by workload
Chat and question answering
Start with Gemma 4 E2B on the CPU path if you have an 8 GB Pi 5 and want a measured baseline. For a HAT-based system, use a currently supported instruction model such as Qwen2.5-Instruct when it is present in the current Hailo catalog. Keep prompts focused and provide local reference material when factual coverage matters.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Coding assistance
Qwen2.5-Coder 1.5B was included in the AI HAT+ 2 launch models and was used for a coding demonstration. It is the most direct HAT candidate in the supplied lineup. On CPU-only hardware, do not assume that a general model will provide the same code quality; test it against the language, libraries and repository structure you use.
Translation
Raspberry Pi demonstrated Qwen2 for questions and translation on the AI HAT+ 2. Gemma 3 1B is the CPU-side model Raspberry Pi explicitly associates with multilingual generation. Compare language pairs and terminology, not just English prompts.
Sentiment, entity extraction and other classifiers
Gemma 3 270M is the strongest fit among the named CPU models because Raspberry Pi describes it for task-specific fine-tuning, including sentiment analysis and entity extraction. Define a strict output format and measure errors on representative samples before deploying it.
Rank #3
- This kit includes an AI HAT+, a metal case and an active cooler. It's compatible with Raspberry Pi 5.
- The Raspberry Pi AI HAT+ features a built-in neural network accelerator, turning your Raspberry Pi 5 into a high-performance, accessible, and power-efficient AI machine.The 13 TOPS variant capably runs neural networks for applications including object detection, semantic and instance segmentation, pose estimation, and more.
- The AI HAT+ communicates using Raspberry Pi 5’s PCIe Gen 3 interface. When the host Raspberry Pi 5 is running an up-to-date Raspberry Pi OS image, it automatically detects the on-board Hailo accelerator and makes the NPU available for AI computing tasks. The built-in rpicam-apps camera applications in Raspberry Pi OS natively support the AI module, automatically using the NPU to run compatible post-processing tasks.
- Conforms to Raspberry Pi HAT+ specification; Supplied with 16mm stacking header, spacers, and screws to enable fitting on Raspberry Pi 5 with Raspberry Pi Active Cooler in place.
- The metal case can protect the Raspberry Pi 5 board from damage, dust and scratches. It can access most ports, including usb-c power jack, micro HDMI ports, usb ports, Ethernet jack, sd card slot, power button and GPIO port.
Document search and retrieval
Use EmbeddingGemma 300M to build a local vector index or retrieval stage. Pair it with a separately selected generator only if the application needs natural-language answers after retrieving relevant passages.
Constrained edge workflows
For a fixed vocabulary, sensor interpretation, routing decision or other bounded task, the smallest model that meets your accuracy target is usually the best engineering choice. A compact model reduces memory pressure and can leave more CPU and RAM for the rest of the application.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Memory, cooling and physical setup
Match model memory to the whole application
Do not allocate all available RAM to model weights. Leave headroom for the operating system, runtime, KV cache, prompt context and companion services. The published 680 MB Gemma 3 270M result leaves considerably more room than the 4,406 MB llama.cpp Gemma 4 E2B result, even though both are described as running on the same Pi 5 benchmark platform.
Plan for sustained workloads
Raspberry Pi’s assembly guidance recommends an Active Cooler on the Pi 5. The AI HAT+ 2 is supplied with its own optional heatsink and mounting hardware. This guidance addresses sustained operation and physical fit; it is not a measured claim that cooling alone changes token speed.
Check mechanical fit before ordering
The HAT, cooler, case and cable arrangement must be compatible. A case designed for a bare Pi may not accommodate the add-on or its heatsink. Treat the mounting hardware and cooling instructions as part of the installation, not as optional cosmetic accessories.
Setup paths
CPU-only setup
- Install a current Raspberry Pi OS image appropriate for your Pi 5 and update the system before adding model software.
- Choose the model by task: Gemma 3 270M for narrow classification or extraction, EmbeddingGemma 300M for vectors and retrieval, Gemma 3 1B for multilingual generation, or Gemma 4 E2B for a measured general-purpose baseline.
- Select a runtime and model build that explicitly supports the model. Keep the quantization and runtime recorded because they materially affect speed and memory.
- Begin with a short context and a small output limit. Confirm that the model loads without exhausting system RAM before increasing context or running concurrent services.
- Measure first-token delay, decode tokens per second, peak memory and output quality using representative prompts.
- Only then integrate the model into a web service, automation or device workflow.
AI HAT+ 2 setup
- Fit the AI HAT+ 2 to the Raspberry Pi 5 using the supplied mounting hardware, and follow Raspberry Pi’s cooling guidance, including an Active Cooler for the Pi 5 where recommended.
- Install the Hailo Gen-AI Model Zoo and the Hailo Ollama server according to the current Raspberry Pi AI software documentation. The documentation gives Debian package 5.1.1 as an installation example; version-specific package names and commands can change.
- Select a model that appears in the current Hailo-compatible catalog. The launch list is historical and should not be treated as the complete inventory.
- Start the local server and use its REST API from your application. Keep the server, model package and runtime versions together in deployment notes.
- Add Open WebUI only if you need a browser interface. Raspberry Pi’s documentation says to run it in Docker because it is incompatible with Python 3.13 as used on Raspberry Pi OS Trixie.
- Test the actual prompt, context length, concurrency and sustained temperature before calling the system production-ready.
Troubleshooting and trade-offs
The model will not load
- Check that the model format and quantization are supported by the selected runtime.
- Compare peak memory with the Pi’s available RAM after the operating system and other services have started.
- Reduce context length and close unrelated services to distinguish a memory problem from a compatibility problem.
- On the HAT route, verify that the model is in the current Hailo catalog rather than relying on an old launch announcement.
Generation is too slow
- Confirm whether the reported number is prefill or decode speed; interactive generation is governed mainly by decode.
- Record thread count, runtime, quantization, prompt size and output length before comparing two builds.
- Try a smaller model when the task permits. Gemma 3 270M’s published 22.58 decode tokens/sec is a useful example of the responsiveness available from a narrow model.
- Check cooling and sustained operation. A short run can look faster than a long workload that shares resources with other services.
Answers are fluent but wrong
- Use a model matched to the task instead of assuming a general chat model is best for extraction or search.
- Supply bounded local context for domain questions; small edge models are not designed to match the knowledge set of large cloud models.
- Evaluate with a fixed test set and inspect failures in the languages, codebase or terminology that matter to your application.
The browser interface fails on Raspberry Pi OS Trixie
Raspberry Pi’s documentation identifies a Python 3.13 incompatibility for Open WebUI on Raspberry Pi OS Trixie and recommends Docker. Use the documented container approach rather than trying to force the interface into the system Python environment.
A practical decision process
- Define the output: conversation, code, translation, embeddings, classification or extraction.
- Choose the smallest model that can meet the quality target: start with Gemma 3 270M or EmbeddingGemma 300M for narrow components.
- Decide whether CPU-only is sufficient: use the Gemma 4 E2B benchmark as a realistic baseline for an 8 GB Pi 5.
- Add the AI HAT+ 2 when supported acceleration, dedicated memory or a Hailo deployment path is worth the hardware and software complexity.
- Validate the exact build: measure quality, decode speed, memory and sustained behavior with your prompts.
- Document versions: model file, quantization, runtime, operating system, Hailo package and server version should be recorded for repeatable deployments.
The best Raspberry Pi 5 language model is therefore the one that clears your task’s quality threshold while leaving enough memory and thermal headroom for the rest of the device. For most first experiments, begin with Gemma 4 E2B or Gemma 3 270M on the CPU; choose the AI HAT+ 2 when you specifically want Raspberry Pi’s supported Hailo acceleration path.
Frequently Asked Questions
Can a Raspberry Pi 5 run a 7B or larger language model?
Raspberry Pi’s guidance recommends caution above 5 billion parameters on a standard Pi. A model’s parameter count is not a compatibility guarantee, so larger models should not be assumed practical without testing the exact runtime, quantization, context and memory use.
Is the older AI HAT+ suitable for local LLMs?
No. Raspberry Pi describes the older AI HAT+ as vision-focused, using Pi system RAM and not supporting LLMs. The AI HAT+ 2 is the Pi 5 add-on documented for local LLM and VLM workloads.
Recommended Free Tools
Which model should I use for embeddings rather than chat?
EmbeddingGemma 300M is the named option for embeddings, retrieval-augmented generation, semantic search and classification. It is not a conversational generator.
Are the published tokens-per-second figures guaranteed on every Pi 5?
No. The figures are for Raspberry Pi’s stated 8 GB Pi 5 setup, four CPU threads, specific token counts, runtimes and quantization/builds. Different RAM tiers, software, prompts and thermals can change the result.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
