OpenClaw + llama.cpp: Your Local AI Assistant, Zero Cloud Dependency

HuggingFace recently published an article about “liberating” OpenClaw from cloud APIs by running models locally. Today I’ll walk you through exactly how I’ve got it running on my own machine — everything local, nothing leaving the box.

What is OpenClaw?

OpenClaw is a personal AI assistant that connects to your favourite messaging channels (Telegram, WhatsApp, Discord, Signal, webchat…) and handles everything from answering questions to running commands on your server, managing your calendar, or checking email.

The interesting part? OpenClaw is model-agnostic. You can use it with GPT-4, Claude, or any OpenAI-compatible endpoint. But the real power comes from running the model locally — no bytes to the cloud, no subscriptions, no rate limits.

Why llama.cpp?

llama.cpp is a C/C++ implementation of the LLaMA-family language models, optimised for both CPU and GPU execution. Here’s why it’s the right tool for the job:

Fully local: everything runs on your machine
Cross-platform: Linux, macOS, Windows
Quantisation support: 35B parameter models in under 20GB of RAM
OpenAI-compatible API: HTTP server exposing the exact same endpoint shape
Open source: no subscriptions, no usage caps

My Setup

I’m running a Qwen3.6-35B-A3B-UD-Q4_K_M.gguf model on my Ubuntu 24.04 desktop. Here’s the full stack:

Building llama.cpp

# Clone the repo
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp

# Build with CUDA support (if you have an NVIDIA GPU)
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j$(nproc)

# Or CPU only
cmake -B build
cmake --build build --config Release -j$(nproc)

Downloading the Model

# Download Qwen3.6-35B in GGUF format (Q4_K_M quantisation)
# Find models on HuggingFace or quantise your own
wget https://huggingface.co/model-name/resolve/main/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf

Starting the Server

./build/bin/llama-server \
  --model Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 8192 \
  --n-gpu-layers 35

The --n-gpu-layers 35 parameter offloads as many layers as possible to the GPU. Skip it entirely for a CPU-only run.

Configuring OpenClaw

Once llama.cpp is running on http://127.0.0.1:8080, wiring up OpenClaw is straightforward:

openclaw onboard \
  --non-interactive \
  --accept-risk \
  --auth-choice custom-api-key \
  --custom-base-url "http://127.0.0.1:8080/v1" \
  --custom-model-id "Qwen3.6-35B-A3B-UD-Q4_K_M.gguf" \
  --custom-api-key "llama.cpp" \
  --secret-input-mode plaintext \
  --custom-compatibility openai

Key points:

--custom-base-url: points to your local llama.cpp server
--custom-model-id: arbitrary name for the model (doesn’t affect functionality)
--custom-api-key: llama.cpp doesn’t require an API key, but the endpoint expects one — anything goes
--custom-compatibility openai: llama.cpp exposes an OpenAI-compatible endpoint

Performance Benchmarks

With my setup (CPU + Q4 quantisation), generation speeds are reasonable for a conversational assistant:

Operation	Approx. Time
Short response (<100 tokens)	2–4 seconds
Medium response (100–500 tokens)	5–15 seconds
Long response (>500 tokens)	15–30 seconds
Cold start (model load)	10–20 seconds

With a dedicated GPU, things get significantly faster. An RTX 3090/4090 (24GB VRAM) can run larger models with much lower latency.

Connecting Messaging Channels

Once OpenClaw is configured, you can wire up whichever channels you want:

openclaw channel add telegram
# Follow the prompts to connect your Telegram bot

openclaw channel add whatsapp
# Scan the QR code with WhatsApp

Webchat

Webchat is available by default at http://localhost:3000 (or whatever port you’ve configured).

Why Go Local?

Total privacy: no data leaves your machine
Zero API costs: no subscriptions, no per-token charges
No rate limits: unlimited requests
Offline-capable: works without internet once the model is loaded
Full customisation: fine-tune with your own data

The Catch

Being honest — it’s not all sunshine.

Speed: on CPU, it’s noticeably slower than a cloud API
Quality: a quantised 35B model isn’t on par with GPT-4o
Resources: you need serious RAM (20GB+ for a 35B model)
Maintenance: you own the infrastructure, for better or worse

Is It Worth It?

Depends on what you’re after:

Privacy and control: absolutely. Best option for staying off Big Tech’s infrastructure.
Speed and top-tier quality: maybe not yet.
Getting started with local AI: excellent starting point. Begin with a smaller model and scale up once you find the right hardware setup.

Next Steps

If you want to go deeper:

Running OpenClaw locally? Drop a comment with your model choice and performance numbers.