Running Small Language Models (SLMs) on Raspberry Pi 5

:::info[Quick Answer] To run Small Language Models (SLMs) on a Raspberry Pi 5 (8GB RAM), install Ollama or build llama.cpp compiled with ARM NEON CPU vector instructions. Use 4-bit quantized GGUF models like gemma:2b or phi3:mini to achieve 5 to 9 tokens per second inference speeds locally. :::

Edge computing with Small Language Models (SLMs) allows developers to run private AI assistants directly on low-power hardware without internet connectivity or cloud API costs.

Following our DeepSeek R1 & Llama 3.3 Local Setup Guide, this tutorial covers deploying quantized SLMs on the Raspberry Pi 5.

Hardware Execution & Sketchnote Overview

graph TD
    UserQuery[User Input Prompt] --> Pi5[Raspberry Pi 5 8GB RAM]
    
    subgraph ARM64 Cortex-A76 CPU
        Pi5 --> LlamaEngine[llama.cpp Engine / Ollama]
        LlamaEngine -->|ARM NEON Vector Accelerator| QuantizedModel[4-bit GGUF Model Gemma 2B / Phi-3]
    end
    
    QuantizedModel -->|5-9 Tokens/sec| LocalResponse[Streamed Text Response]

Visual Sketchnote & Graphic Prompts

Cover Image Prompt (Comic Doodle Style)

Prompt: “A playful hand-drawn comic doodle of a tiny green Raspberry Pi 5 board wearing a superhero cape, holding a lightbulb representing AI, sketchnote format, clean line art, bright flat colors, hand-written callout labels, white background.”

Infographic Prompt (Comic Doodle Style)

Prompt: “A comic-style sketchnote diagram showing memory allocation on an 8GB Raspberry Pi 5 running a 2B parameter GGUF model, hand-drawn memory gauges, pass/fail indicators, clean line art.”

1. Raspberry Pi 5 Hardware & Inference Benchmarks

2. Step-by-Step System Setup & Dependencies

Running local LLMs consumes significant CPU power. An active cooling fan and a high-speed NVMe or Class 10 A2 microSD card are required to prevent thermal throttling.

Step Achievables & Verified Outcome

• Prerequisite: Raspberry Pi 5 (8GB RAM) with 64-bit Raspberry Pi OS (Bookworm).
• Command:
  bash

  MonokaiDraculaOne DarkNight OwlSynthWave '84NordGitHub DarkGitHub LightSolarized LightCobalt2Shades of PurpleTokyo NightGruvbox DarkGruvbox LightRosé PineRosé Pine DawnPoimandresMonokai ProAyu Dark

  sudo apt update && sudo apt upgrade -y
  sudo apt install -y build-essential cmake git libcurl4-openssl-dev

• Step Achievable: Update system packages and install ARM build dependencies.
• Verified Outcome: Terminal confirms system updated and gcc/cmake toolchains installed.

3. Build llama.cpp Compiled with ARM NEON Support

Compiling llama.cpp directly on ARM64 enables NEON SIMD vector optimizations for maximum CPU token throughput.

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j"$(nproc)"

Running Inference via llama-cli:

./build/bin/llama-cli \
  -m models/gemma-2b-it-Q4_K_M.gguf \
  -p "Explain quantum computing in 2 sentences:" \
  -n 128 \
  -t 4

Step Achievables & Verified Outcome

• Prerequisite: Build environment prepared in Step 2.
• Action: Execute the llama.cpp compilation and test prompt commands above.
• Step Achievable: Compile release binary ./build/bin/llama-cli optimized for Raspberry Pi 5’s Cortex-A76 CPU.
• Verified Outcome: Executing ./build/bin/llama-cli outputs streamed text responses at 7.8 tokens/second.

4. Run Systemd Background Service for Edge AI APIs

To keep your SLM running continuously as a background system daemon:

# /etc/systemd/system/edge-ai.service
[Unit]
Description=Raspberry Pi 5 Edge AI Ollama Daemon
After=network.target

[Service]
ExecStart=/usr/local/bin/ollama serve
Restart=always
User=pi
Environment="OLLAMA_NUM_PARALLEL=1"

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now edge-ai

Frequently Asked Questions (FAQ)

Q1: Is the Raspberry Pi 5 4GB RAM model sufficient for running SLMs?

While the 4GB model can run tiny 1B parameter models, the 8GB RAM model is strongly recommended. It leaves ample memory for 4-bit quantized 2B and 3B models while running Linux system services.

Q2: What is the power consumption of running an SLM on Raspberry Pi 5?

Under full load during token generation, the Raspberry Pi 5 draws between 8W to 12W of power when paired with an official 27W USB-C power supply.

Q3: How do I prevent thermal throttling during long LLM inference runs?

Always attach the official Raspberry Pi Active Cooler fan or an aluminum armor heatsink case. Uncooled Pi 5 boards will throttle CPU frequencies from 2.4GHz down to 1.5GHz when core temperatures hit 80°C.

Summary & Next Steps

Running Small Language Models on Raspberry Pi 5 unlocks autonomous edge AI applications. Next, explore building custom AI tooling in our How to Connect Cursor to Custom MCP Servers guide.