GOAL.md

# GOAL.md — HF PRO ZeroGPU Engine, Training Lab & OmniRoute Provider

**Owner:** abalanescu
**Repo:** `abalanescu/flow` (HF Space, ZeroGPU)
**Mission:** Maximize the Hugging Face PRO subscription without confusing included inference with separately billed compute. Build a reliable private LLM/audio/embeddings service, connect it to OmniRoute, and use HF Hub/Spaces/Datasets/AutoTrain or rented hardware for reproducible training experiments.

---

## ZeroGPU Capability & Model Limits (Skeptic Check)

| Resource Category | Supported Models / Specs | Status / Feasibility |
|---|---|---|
| **Max VRAM / GPU** | Nvidia A100 (80GB VRAM) / A10G (24GB) | Max model size: ~70B quantized (Q4_K_M ~40GB VRAM) |
| **LLMs (Custom & Uncensored)** | Gemma 4 26B Heretic, Qwen 2.5 32B, DeepSeek R1 Distill 32B, Llama 3.3 70B Q4 | ✅ **High Feasibility** (Gemma 26B live & verified) |
| **DeepSeek V3 / V4 (671B)** | Full 671B MoE model | ❌ **NOT Possible** (Requires 8x H100s, 700GB+ VRAM) |
| **DeepSeek R1 Distills** | DeepSeek-R1-Distill-Qwen-32B / 14B | ✅ **Supported** |
| **Audio STT (Transcribe)** | Local Whisper Large v3 on ZeroGPU | ✅ **Supported** (replaces serverless API limit) |
| **Audio TTS (Voice Synth)** | Local Kokoro-82M / F5-TTS on ZeroGPU | ✅ **Supported** |
| **Embeddings** | BGE-M3, Qwen2.5-Coder-Embedding | ✅ **Supported** (Fast batch vector search) |

---

## Milestone 1 — Foundation & Live Verification (DONE)
- [x] Wire ZeroGPU Gemma Heretic chat (local GGUF at `/data`).
- [x] Deploy Space (`abalanescu-flow.hf.space`) with Gradio 5.28.0.
- [x] Confirm ZeroGPU Gemma 26B chat runs live on A100 (`duration=120s`).

## Milestone 2 — Private LLM Provider & Multi-Model Engine (DONE, live-verified)
- [x] Integrate `unsloth/Qwen3.8-27B-GGUF` (`Qwen3.8-27B-Q6_K.gguf`) with lazy background download to `/data`.
- [x] Remove artificial context caps (`n_ctx=0` uses GGUF native 262,144 tokens).
- [x] Implement OpenAI-compatible endpoints (`/v1/models`, `/v1/chat/completions`) with bearer authentication.
- [x] Live-verify OpenAI chat completions on ZeroGPU for both Gemma and Qwen 3.8 models.
- [ ] **Remove Image Gen:** Drop FLUX tab to focus 100% on LLMs, Audio, and Embeddings.
- [ ] **ZeroGPU Local STT (Whisper Large v3):** Move Whisper from paid serverless API to ZeroGPU local model.
- [ ] **ZeroGPU Local TTS (Kokoro/F5-TTS):** Add local high-speed voice synthesis on ZeroGPU.
- [ ] **ZeroGPU Embeddings Endpoint:** Add BGE-M3 / Qwen vector embedding tab & API.
- [x] **Private OpenAI-Compatible API:** Expose authenticated `/v1/chat/completions` and `/v1/models` for OmniRoute/Open-WebUI integration.
- [ ] Add `/v1/embeddings` after selecting and validating a dedicated embedding model.

## Milestone 3 — HF PRO Value Audit (IN PROGRESS)
- [ ] Inventory account entitlements and actual limits from the HF billing/settings pages.
- [ ] Measure ZeroGPU queue time, GPU seconds, cold start, throughput, and daily quota consumption.
- [ ] Inventory PRO storage, private Spaces, private model/dataset repos, collaboration, secrets, buckets, and Inference Providers credits.
- [ ] Separate free/included features from usage-based billing. No claim is DONE without a live observation or official documentation link.

## Milestone 4 — Training & Dataset Lab
- [ ] Build a small private dataset pipeline: collect, clean, deduplicate, redact, split, and version on HF Datasets.
- [ ] Run a cheap supervised fine-tuning/LoRA pilot on a 1B–8B model using AutoTrain or rented GPU. ZeroGPU is for inference, not dependable training.
- [ ] Evaluate base vs adapter with a fixed holdout set and publish private model cards, metrics, and rollback instructions.
- [ ] Decide whether training is worth the cost before scaling beyond a pilot.

## Milestone 5 — OmniRoute Provider & Automation
- [ ] Add the Space as a named OmniRoute provider with the exact model IDs returned by `/v1/models`.
- [ ] Configure long timeout, concurrency=1 initially, retry only on transport failures, and no blind fallback replay for GPU jobs.
- [ ] Add health/model discovery and a smoke test to the homelab deployment runbook.
- [ ] Add local STT, TTS, and embeddings as separate providers only after each has a verified endpoint.
- [ ] Build document summarization, meeting transcription, semantic search, and structured extraction workflows.

## Non-goals / guardrails
- ZeroGPU is not a guaranteed training cluster. Use it for inference and short GPU functions.
- Do not advertise Qwen 3.6, DeepSeek V4, or any model until the exact repository, license, format, VRAM, and live test are confirmed.
- Do not claim `/v1/embeddings`, STT, or TTS exists until a curl test returns a real response.
- Never put `HF_TOKEN` or `FLOW_API_KEY` in git, docs, curl history, or OmniRoute source files.