GOAL.md

GOAL.md — HF PRO ZeroGPU Engine, Training Lab & OmniRoute Provider

Owner: abalanescu Repo: abalanescu/flow (HF Space, ZeroGPU) Mission: Maximize the Hugging Face PRO subscription without confusing included inference with separately billed compute. Build a reliable private LLM/audio/embeddings service, connect it to OmniRoute, and use HF Hub/Spaces/Datasets/AutoTrain or rented hardware for reproducible training experiments.

---

ZeroGPU Capability & Model Limits (Skeptic Check)

Resource Category	Supported Models / Specs	Status / Feasibility
Max VRAM / GPU	Nvidia A100 (80GB VRAM) / A10G (24GB)	Max model size: ~70B quantized (Q4_K_M ~40GB VRAM)
LLMs (Custom & Uncensored)	Gemma 4 26B Heretic, Qwen 2.5 32B, DeepSeek R1 Distill 32B, Llama 3.3 70B Q4	✅ High Feasibility (Gemma 26B live & verified)
DeepSeek V3 / V4 (671B)	Full 671B MoE model	❌ NOT Possible (Requires 8x H100s, 700GB+ VRAM)
DeepSeek R1 Distills	DeepSeek-R1-Distill-Qwen-32B / 14B	✅ Supported
Audio STT (Transcribe)	Local Whisper Large v3 on ZeroGPU	✅ Supported (replaces serverless API limit)
Audio TTS (Voice Synth)	Local Kokoro-82M / F5-TTS on ZeroGPU	✅ Supported
Embeddings	BGE-M3, Qwen2.5-Coder-Embedding	✅ Supported (Fast batch vector search)

---

Milestone 1 — Foundation & Live Verification (DONE)

• Wire ZeroGPU Gemma Heretic chat (local GGUF at /data).
• Deploy Space (abalanescu-flow.hf.space) with Gradio 5.28.0.
• Confirm ZeroGPU Gemma 26B chat runs live on A100 (duration=120s).

Milestone 2 — Private LLM Provider & Multi-Model Engine (DONE, live-verified)

• Integrate unsloth/Qwen3.8-27B-GGUF (Qwen3.8-27B-Q6_K.gguf) with lazy background download to /data.
• Remove artificial context caps (n_ctx=0 uses GGUF native 262,144 tokens).
• Implement OpenAI-compatible endpoints (/v1/models, /v1/chat/completions) with bearer authentication.
• Live-verify OpenAI chat completions on ZeroGPU for both Gemma and Qwen 3.8 models.
• Remove Image Gen: Drop FLUX tab to focus 100% on LLMs, Audio, and Embeddings.
• ZeroGPU Local STT (Whisper Large v3): Move Whisper from paid serverless API to ZeroGPU local model.
• ZeroGPU Local TTS (Kokoro/F5-TTS): Add local high-speed voice synthesis on ZeroGPU.
• ZeroGPU Embeddings Endpoint: Add BGE-M3 / Qwen vector embedding tab & API.
• Private OpenAI-Compatible API: Expose authenticated /v1/chat/completions and /v1/models for OmniRoute/Open-WebUI integration.
• Add /v1/embeddings after selecting and validating a dedicated embedding model.

Milestone 3 — HF PRO Value Audit (IN PROGRESS)

• Inventory account entitlements and actual limits from the HF billing/settings pages.
• Measure ZeroGPU queue time, GPU seconds, cold start, throughput, and daily quota consumption.
• Inventory PRO storage, private Spaces, private model/dataset repos, collaboration, secrets, buckets, and Inference Providers credits.
• Separate free/included features from usage-based billing. No claim is DONE without a live observation or official documentation link.

Milestone 4 — Training & Dataset Lab

• Build a small private dataset pipeline: collect, clean, deduplicate, redact, split, and version on HF Datasets.
• Run a cheap supervised fine-tuning/LoRA pilot on a 1B–8B model using AutoTrain or rented GPU. ZeroGPU is for inference, not dependable training.
• Evaluate base vs adapter with a fixed holdout set and publish private model cards, metrics, and rollback instructions.
• Decide whether training is worth the cost before scaling beyond a pilot.

Milestone 5 — OmniRoute Provider & Automation

• Add the Space as a named OmniRoute provider with the exact model IDs returned by /v1/models.
• Configure long timeout, concurrency=1 initially, retry only on transport failures, and no blind fallback replay for GPU jobs.
• Add health/model discovery and a smoke test to the homelab deployment runbook.
• Add local STT, TTS, and embeddings as separate providers only after each has a verified endpoint.
• Build document summarization, meeting transcription, semantic search, and structured extraction workflows.

Non-goals / guardrails

• ZeroGPU is not a guaranteed training cluster. Use it for inference and short GPU functions.
• Do not advertise Qwen 3.6, DeepSeek V4, or any model until the exact repository, license, format, VRAM, and live test are confirmed.
• Do not claim /v1/embeddings, STT, or TTS exists until a curl test returns a real response.
• Never put HF_TOKEN or FLOW_API_KEY in git, docs, curl history, or OmniRoute source files.