hf-space-rag/README.md

---
title: CPU RAG Space
emoji: 🦅
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit
---

# CPU RAG Space — Qwen2.5-1.5B + FAISS (free tier, no GPU)

A self-contained Retrieval-Augmented Generation service that runs **entirely on
CPU** and fits the Hugging Face **free tier** (2 vCPU / 16 GB).

| Component | What | Why |
|-----------|------|-----|
| Embeddings | `BAAI/bge-small-en-v1.5` via `fastembed` (ONNX) | fast on CPU, no PyTorch, ~130 MB |
| Vector DB | **FAISS** (in-memory) | tiny, instant search |
| LLM | `Qwen2.5-1.5B-Instruct` GGUF Q4_K_M via llama.cpp | fast on CPU (~12–20 tok/s), ~1 GB |
| API | OpenAI-compatible `/v1/chat/completions` + web UI | drop-in for any client |

Total footprint ≈ 1.5–2 GB RAM. Retrieval adds ~20 ms; the LLM is the only
real latency.

## Deploy (drag & drop)

1. Create a new Space → **Docker** (blank template).
2. Drag **all files in this folder** into the Space repo (keep the structure —
   `documents/` included).
3. Push. First build takes a few minutes (it bakes the ~1 GB LLM and the
   embedder into the image so cold starts are instant).

## Use it

Web UI: open the Space URL. Upload `.txt`/`.md` files and ask questions.

API (OpenAI-compatible):

```python
from openai import OpenAI
client = OpenAI(base_url="https://<user>-<space>.hf.space/v1", api_key="x")
r = client.chat.completions.create(
    model="cpu-rag",
    messages=[{"role": "user", "content": "How does retrieval work here?"}],
)
print(r.choices[0].message.content)
```

Extra endpoints: `POST /ingest` (upload a doc), `GET /stats`.

## Add your own knowledge

- Put `.txt`/`.md` files in `documents/` before pushing (indexed at startup), or
- Upload them at runtime via the UI / `POST /ingest`.

> Note: the free tier has ephemeral storage, so runtime-uploaded docs are lost
> on restart. For a permanent corpus, commit files into `documents/`.

## Swap the model

Change these in the `Dockerfile` (confirm exact filenames on the repo's *Files*
tab):

- Faster / smaller: `Qwen/Qwen2.5-0.5B-Instruct-GGUF`
- Coding-focused: `Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF`