---
title: CPU RAG Space
emoji: 🦅
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit
---

# CPU RAG Space — Qwen3.5-0.8B + FAISS (free tier, no GPU)

A self-contained Retrieval-Augmented Generation service that runs **entirely on
CPU** and fits the Hugging Face **free tier** (2 vCPU / 16 GB).

| Component | What | Why |
|-----------|------|-----|
| Embeddings | `BAAI/bge-small-en-v1.5` via `fastembed` (ONNX) | fast on CPU, no PyTorch, ~130 MB |
| Vector DB | **FAISS** (in-memory) | tiny, instant search |
| LLM | `Qwen3.5-0.8B` (MoE) GGUF Q4_K_M via llama.cpp | ~533 MB, small MoE — fast on 2 vCPU, smarter than a dense 0.5B |
| API | OpenAI-compatible `/v1/chat/completions` + web UI | drop-in for any client |

Total footprint ≈ 0.8–1.1 GB RAM. Retrieval adds ~20 ms; the LLM is the only
real latency.

> **Runtime requirement:** Qwen3.5 uses the `qwen35` architecture, so this needs
> **`llama-cpp-python >= 0.3.32`** (llama.cpp ≥ b9616). The Dockerfile builds it
> from source. Older pins fail with *"unknown architecture 'qwen35'"*.

**Tuned for CPU speed:** 2K context (small KV cache), flash-attention, short
256-token answers, relevance-gated RAG (greetings/off-topic skip retrieval), a
startup warm-up, and a **streaming** web UI so the first word appears in ~1 s.
See `.github/workflows/keepalive.yml` for the keep-warm ping.

## Deploy (drag & drop)

1. Create a new Space → **Docker** (blank template).
2. Drag **all files in this folder** into the Space repo (keep the structure —
   `documents/` included).
3. Push. First build takes a few minutes (it bakes the ~1 GB LLM and the
   embedder into the image so cold starts are instant).

## Use it

Web UI: open the Space URL. Upload `.txt`/`.md` files and ask questions.

API (OpenAI-compatible):

```python
from openai import OpenAI
client = OpenAI(base_url="https://<user>-<space>.hf.space/v1", api_key="x")
r = client.chat.completions.create(
    model="cpu-rag",
    messages=[{"role": "user", "content": "How does retrieval work here?"}],
)
print(r.choices[0].message.content)
```

Extra endpoints: `POST /ingest` (upload a doc), `GET /stats`.

## Add your own knowledge

- Put `.txt`/`.md` files in `documents/` before pushing (indexed at startup), or
- Upload them at runtime via the UI / `POST /ingest`.

> Note: the free tier has ephemeral storage, so runtime-uploaded docs are lost
> on restart. For a permanent corpus, commit files into `documents/`.

## Swap the model

Change `LLM_REPO` / `LLM_FILE` in the `Dockerfile` (confirm exact filenames on
the repo's *Files* tab):

- Current (small MoE, fast on CPU): `unsloth/Qwen3.5-0.8B-GGUF` → `Qwen3.5-0.8B-Q4_K_M.gguf`
- Fastest tiny dense: `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF`
- Best general quality (~½ speed): `HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF`

### MTP (Multi-Token Prediction) — later

`unsloth/Qwen3.5-0.8B-MTP-GGUF` adds self-speculative decoding for a ~1.4–1.7×
decode speedup. It's **not** a drop-in here: the MTP flags (`--spec-type
draft-mtp`) live in **`llama-server`**, not the `llama_cpp.Llama()` API this app
uses. To try it, run `llama-server` as the backend and proxy to it. Feasible on
CPU only because the model is tiny (an MTP 9B on 2 vCPU stays ~1.5–2.5 tok/s).

## Keep it warm (avoid cold starts)

Free Spaces sleep when idle, so the first request after a nap pays a ~25 s wake.
`.github/workflows/keepalive.yml` pings `/stats` every 10 minutes from GitHub's
runners to keep the Space (and the loaded model) warm — no cost, fully external.
GitHub disables scheduled workflows after 60 days of repo inactivity; push any
commit to re-arm it.