README.md

---
title: Vaaani Flagship CPU
emoji: 🌉
colorFrom: indigo
colorTo: green
sdk: docker
app_port: 7860
pinned: false
---

# Vaaani Flagship — CPU inference (free tier)

Serves the Vaaani curriculum tutor (Qwen2.5-3B base GGUF + the Root-Bridge / sound-spiral
LoRA) entirely on CPU via `llama-cpp-python`. No GPU. The CPU-only design is the moat —
and it is exactly what lets this run on Hugging Face's **free** 2-vCPU / 16-GB Space.

## Endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | `/` | health (returns instantly; model loads lazily on first chat) |
| POST | `/chat` | `{ "messages": [...] }` → `{ "reply": "..." }` |
| POST | `/chat/stream` | Server-Sent Events, token-by-token (use this in the UI) |
| POST | `/v1/chat/completions` | OpenAI-compatible drop-in for the gateway |

## Configuration (Space → Settings → Variables, not Secrets)
| Var | Default | Notes |
|---|---|---|
| `VAAANI_MODEL_REPO` | `Shaankar39/vaaani-flagship-gguf` | HF **model** repo holding the GGUF files |
| `VAAANI_BASE_FILE` | `vaaani-base-q4_k_m.gguf` | Q4_K_M for free CPU; swap to Q8 on a paid Space |
| `VAAANI_LORA_FILE` | `vaaani-flagship-lora-f16.gguf` | the curriculum adapter |
| `N_THREADS` | `2` | = the free tier's 2 vCPUs |
| `N_CTX` | `2048` | curriculum prompts are short; keep it small for speed |
| `MAX_TOKENS` | `512` | lesson turns are brief |

## How the GGUF gets here
The Space does **not** bundle the 2–3 GB model. It pulls it at first request from the HF
model repo above (`hf_hub_download`). So: after training, upload the two GGUF files to that
model repo once, and this Space serves them. Swapping quant = change one Variable, no rebuild.

## Free-tier reality (honest)
- **Accuracy:** identical to local — same weights; Q4_K_M ≈ Q8 for this scripted task.
- **Speed:** a few tok/s on 2 vCPU (~10–25 s/turn). Streaming hides most of it.
- **Sleep:** free Spaces idle-sleep after 48 h → ~30–60 s cold start on the next hit.
- **Scale:** one request at a time. For concurrent classrooms, upgrade to a paid **CPU**
  Space (never a GPU) — funded from pilot revenue, not upfront.