README.md

metadata

title: CPU RAG Space
emoji: 🦅
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit

CPU RAG Space — Qwen3.5-0.8B + FAISS (free tier, no GPU)

A self-contained Retrieval-Augmented Generation service that runs entirely on CPU and fits the Hugging Face free tier (2 vCPU / 16 GB).

Component	What	Why
Embeddings	BAAI/bge-small-en-v1.5 via fastembed (ONNX)	fast on CPU, no PyTorch, ~130 MB
Vector DB	FAISS (in-memory)	tiny, instant search
LLM	Qwen3.5-0.8B (MoE) GGUF Q4_K_M via llama.cpp	~533 MB, small MoE — fast on 2 vCPU, smarter than a dense 0.5B
API	OpenAI-compatible /v1/chat/completions + web UI	drop-in for any client

Total footprint ≈ 0.8–1.1 GB RAM. Retrieval adds ~20 ms; the LLM is the only real latency.

Runtime requirement: Qwen3.5 uses the qwen35 architecture, so this needs llama-cpp-python >= 0.3.32 (llama.cpp ≥ b9616). The Dockerfile builds it from source. Older pins fail with "unknown architecture 'qwen35'".

Tuned for CPU speed: 2K context (small KV cache), flash-attention, short 256-token answers, relevance-gated RAG (greetings/off-topic skip retrieval), a startup warm-up, and a streaming web UI so the first word appears in ~1 s. See .github/workflows/keepalive.yml for the keep-warm ping.

Deploy (drag & drop)

1. Create a new Space → Docker (blank template).
2. Drag all files in this folder into the Space repo (keep the structure — documents/ included).
3. Push. First build takes a few minutes (it bakes the ~1 GB LLM and the embedder into the image so cold starts are instant).

Use it

Web UI: open the Space URL. Upload .txt/.md files and ask questions.

API (OpenAI-compatible):

from openai import OpenAI
client = OpenAI(base_url="https://<user>-<space>.hf.space/v1", api_key="x")
r = client.chat.completions.create(
    model="cpu-rag",
    messages=[{"role": "user", "content": "How does retrieval work here?"}],
)
print(r.choices[0].message.content)

Extra endpoints: POST /ingest (upload a doc), GET /stats.

Add your own knowledge

• Put .txt/.md files in documents/ before pushing (indexed at startup), or
• Upload them at runtime via the UI / POST /ingest.

Note: the free tier has ephemeral storage, so runtime-uploaded docs are lost on restart. For a permanent corpus, commit files into documents/.

Swap the model

Change LLM_REPO / LLM_FILE in the Dockerfile (confirm exact filenames on the repo's Files tab):

• Current (small MoE, fast on CPU): unsloth/Qwen3.5-0.8B-GGUF → Qwen3.5-0.8B-Q4_K_M.gguf
• Fastest tiny dense: Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF
• Best general quality (~½ speed): HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF

MTP (Multi-Token Prediction) — later

unsloth/Qwen3.5-0.8B-MTP-GGUF adds self-speculative decoding for a ~1.4–1.7× decode speedup. It's not a drop-in here: the MTP flags (--spec-type draft-mtp) live in llama-server, not the llama_cpp.Llama() API this app uses. To try it, run llama-server as the backend and proxy to it. Feasible on CPU only because the model is tiny (an MTP 9B on 2 vCPU stays ~1.5–2.5 tok/s).

Keep it warm (avoid cold starts)

Free Spaces sleep when idle, so the first request after a nap pays a ~25 s wake. .github/workflows/keepalive.yml pings /stats every 10 minutes from GitHub's runners to keep the Space (and the loaded model) warm — no cost, fully external. GitHub disables scheduled workflows after 60 days of repo inactivity; push any commit to re-arm it.