README.md

metadata

title: Vaaani Flagship CPU
emoji: 🌉
colorFrom: indigo
colorTo: green
sdk: docker
app_port: 7860
pinned: false

Vaaani Flagship — CPU inference (free tier)

Serves the Vaaani curriculum tutor (Qwen2.5-3B base GGUF + the Root-Bridge / sound-spiral LoRA) entirely on CPU via llama-cpp-python. No GPU. The CPU-only design is the moat — and it is exactly what lets this run on Hugging Face's free 2-vCPU / 16-GB Space.

Endpoints

Method	Path	Purpose
GET	/	health (returns instantly; model loads lazily on first chat)
POST	/chat	{ "messages": [...] } → { "reply": "..." }
POST	/chat/stream	Server-Sent Events, token-by-token (use this in the UI)
POST	/v1/chat/completions	OpenAI-compatible drop-in for the gateway

Configuration (Space → Settings → Variables, not Secrets)

Var	Default	Notes
VAAANI_MODEL_REPO	Shaankar39/vaaani-flagship-gguf	HF model repo holding the GGUF files
VAAANI_BASE_FILE	vaaani-base-q4_k_m.gguf	Q4_K_M for free CPU; swap to Q8 on a paid Space
VAAANI_LORA_FILE	vaaani-flagship-lora-f16.gguf	the curriculum adapter
N_THREADS	2	= the free tier's 2 vCPUs
N_CTX	2048	curriculum prompts are short; keep it small for speed
MAX_TOKENS	512	lesson turns are brief

How the GGUF gets here

The Space does not bundle the 2–3 GB model. It pulls it at first request from the HF model repo above (hf_hub_download). So: after training, upload the two GGUF files to that model repo once, and this Space serves them. Swapping quant = change one Variable, no rebuild.

Free-tier reality (honest)

• Accuracy: identical to local — same weights; Q4_K_M ≈ Q8 for this scripted task.
• Speed: a few tok/s on 2 vCPU (~10–25 s/turn). Streaming hides most of it.
• Sleep: free Spaces idle-sleep after 48 h → ~30–60 s cold start on the next hit.
• Scale: one request at a time. For concurrent classrooms, upgrade to a paid CPU Space (never a GPU) — funded from pilot revenue, not upfront.