hf-space-rag/README.md

metadata

title: CPU RAG Space
emoji: 🦅
colorFrom: indigo
colorTo: purple
sdk: docker
app_port: 7860
pinned: false
license: mit

CPU RAG Space — Qwen2.5-1.5B + FAISS (free tier, no GPU)

A self-contained Retrieval-Augmented Generation service that runs entirely on CPU and fits the Hugging Face free tier (2 vCPU / 16 GB).

Component	What	Why
Embeddings	BAAI/bge-small-en-v1.5 via fastembed (ONNX)	fast on CPU, no PyTorch, ~130 MB
Vector DB	FAISS (in-memory)	tiny, instant search
LLM	Qwen2.5-1.5B-Instruct GGUF Q4_K_M via llama.cpp	fast on CPU (~12–20 tok/s), ~1 GB
API	OpenAI-compatible /v1/chat/completions + web UI	drop-in for any client

Total footprint ≈ 1.5–2 GB RAM. Retrieval adds ~20 ms; the LLM is the only real latency.

Deploy (drag & drop)

1. Create a new Space → Docker (blank template).
2. Drag all files in this folder into the Space repo (keep the structure — documents/ included).
3. Push. First build takes a few minutes (it bakes the ~1 GB LLM and the embedder into the image so cold starts are instant).

Use it

Web UI: open the Space URL. Upload .txt/.md files and ask questions.

API (OpenAI-compatible):

from openai import OpenAI
client = OpenAI(base_url="https://<user>-<space>.hf.space/v1", api_key="x")
r = client.chat.completions.create(
    model="cpu-rag",
    messages=[{"role": "user", "content": "How does retrieval work here?"}],
)
print(r.choices[0].message.content)

Extra endpoints: POST /ingest (upload a doc), GET /stats.

Add your own knowledge

• Put .txt/.md files in documents/ before pushing (indexed at startup), or
• Upload them at runtime via the UI / POST /ingest.

Note: the free tier has ephemeral storage, so runtime-uploaded docs are lost on restart. For a permanent corpus, commit files into documents/.

Swap the model

Change these in the Dockerfile (confirm exact filenames on the repo's Files tab):

• Faster / smaller: Qwen/Qwen2.5-0.5B-Instruct-GGUF
• Coding-focused: Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF