Spaces:
Build error
Build error
| title: CPU RAG Space | |
| emoji: 🦅 | |
| colorFrom: indigo | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| license: mit | |
| # CPU RAG Space — Qwen3.5-0.8B + FAISS (free tier, no GPU) | |
| A self-contained Retrieval-Augmented Generation service that runs **entirely on | |
| CPU** and fits the Hugging Face **free tier** (2 vCPU / 16 GB). | |
| | Component | What | Why | | |
| |-----------|------|-----| | |
| | Embeddings | `BAAI/bge-small-en-v1.5` via `fastembed` (ONNX) | fast on CPU, no PyTorch, ~130 MB | | |
| | Vector DB | **FAISS** (in-memory) | tiny, instant search | | |
| | LLM | `Qwen3.5-0.8B` (MoE) GGUF Q4_K_M via llama.cpp | ~533 MB, small MoE — fast on 2 vCPU, smarter than a dense 0.5B | | |
| | API | OpenAI-compatible `/v1/chat/completions` + web UI | drop-in for any client | | |
| Total footprint ≈ 0.8–1.1 GB RAM. Retrieval adds ~20 ms; the LLM is the only | |
| real latency. | |
| > **Runtime requirement:** Qwen3.5 uses the `qwen35` architecture, so this needs | |
| > **`llama-cpp-python >= 0.3.32`** (llama.cpp ≥ b9616). The Dockerfile builds it | |
| > from source. Older pins fail with *"unknown architecture 'qwen35'"*. | |
| **Tuned for CPU speed:** 2K context (small KV cache), flash-attention, short | |
| 256-token answers, relevance-gated RAG (greetings/off-topic skip retrieval), a | |
| startup warm-up, and a **streaming** web UI so the first word appears in ~1 s. | |
| See `.github/workflows/keepalive.yml` for the keep-warm ping. | |
| ## Deploy (drag & drop) | |
| 1. Create a new Space → **Docker** (blank template). | |
| 2. Drag **all files in this folder** into the Space repo (keep the structure — | |
| `documents/` included). | |
| 3. Push. First build takes a few minutes (it bakes the ~1 GB LLM and the | |
| embedder into the image so cold starts are instant). | |
| ## Use it | |
| Web UI: open the Space URL. Upload `.txt`/`.md` files and ask questions. | |
| API (OpenAI-compatible): | |
| ```python | |
| from openai import OpenAI | |
| client = OpenAI(base_url="https://<user>-<space>.hf.space/v1", api_key="x") | |
| r = client.chat.completions.create( | |
| model="cpu-rag", | |
| messages=[{"role": "user", "content": "How does retrieval work here?"}], | |
| ) | |
| print(r.choices[0].message.content) | |
| ``` | |
| Extra endpoints: `POST /ingest` (upload a doc), `GET /stats`. | |
| ## Add your own knowledge | |
| - Put `.txt`/`.md` files in `documents/` before pushing (indexed at startup), or | |
| - Upload them at runtime via the UI / `POST /ingest`. | |
| > Note: the free tier has ephemeral storage, so runtime-uploaded docs are lost | |
| > on restart. For a permanent corpus, commit files into `documents/`. | |
| ## Swap the model | |
| Change `LLM_REPO` / `LLM_FILE` in the `Dockerfile` (confirm exact filenames on | |
| the repo's *Files* tab): | |
| - Current (small MoE, fast on CPU): `unsloth/Qwen3.5-0.8B-GGUF` → `Qwen3.5-0.8B-Q4_K_M.gguf` | |
| - Fastest tiny dense: `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF` | |
| - Best general quality (~½ speed): `HuggingFaceTB/SmolLM2-1.7B-Instruct-GGUF` | |
| ### MTP (Multi-Token Prediction) — later | |
| `unsloth/Qwen3.5-0.8B-MTP-GGUF` adds self-speculative decoding for a ~1.4–1.7× | |
| decode speedup. It's **not** a drop-in here: the MTP flags (`--spec-type | |
| draft-mtp`) live in **`llama-server`**, not the `llama_cpp.Llama()` API this app | |
| uses. To try it, run `llama-server` as the backend and proxy to it. Feasible on | |
| CPU only because the model is tiny (an MTP 9B on 2 vCPU stays ~1.5–2.5 tok/s). | |
| ## Keep it warm (avoid cold starts) | |
| Free Spaces sleep when idle, so the first request after a nap pays a ~25 s wake. | |
| `.github/workflows/keepalive.yml` pings `/stats` every 10 minutes from GitHub's | |
| runners to keep the Space (and the loaded model) warm — no cost, fully external. | |
| GitHub disables scheduled workflows after 60 days of repo inactivity; push any | |
| commit to re-arm it. | |