docs/HF_PRO_VALUE_PLAN.md

HF PRO Value Plan

What we know from this project

Capability	Current evidence	Intended use
Private Space	abalanescu/flow is private and deployed	Host the private provider UI/API
ZeroGPU (PRO)	Gemma 26B and Qwen 27B Q6 returned live responses	Multi-model private OpenAI upstream
Bucket mount	GGUF files are mounted under /data	Keep 23GB+ model weights persistent across restarts
Dynamic Hardware	A10G (24GB) / RTX Pro 6000 (48GB large) verified	Run quantized 26B–27B models on-demand
Inference Providers	Previous FLUX/Whisper calls returned 402	Treat credit as scarce, optional spend

Verified ZeroGPU Architecture & Sizing Rules

1. Hardware & VRAM:

   • zero-a10g assigns 24GB VRAM (A10G) by default; new ZeroGPU infrastructure supports large (48GB) and xlarge (96GB, 2x quota cost).
   • Model weights vs KV-cache: For a ~22.9 GB model (Qwen 27B Q6_K) on 24GB VRAM, remaining VRAM is ~1.1 GB.
   • Context sizing: n_ctx=8192 safely fits KV-cache in memory. n_ctx=0 attempts to pre-allocate full 262k context (~16GB KV cache), which triggers CUDA OOM during context initialization.

2. ZeroGPU Supervisor Integration:

   • Handlers must be registered through Gradio's event graph at demo.launch(prevent_thread_lock=True, ssr_mode=False).
   • OpenAI routes (/v1/models, /v1/chat/completions) mount onto demo.app.add_api_route(...) to share the single listener on port 7860 without conflicting with Space supervisor probes.

3. Persistent Bucket Storage:

   • GGUF weights stored in /data avoid cold-start download latency.
   • Uncached models use hf_hub_download into /data on first request.

What must be measured, not assumed

HF plan benefits and quotas change. Verify them in the account UI and official docs before budgeting:

1. ZeroGPU daily quota, queue priority, duration limits, hardware actually assigned, and whether quota is per-user or per-Space.
2. Private Space count, hardware eligibility, secrets, storage, buckets, and bandwidth.
3. Private model and dataset repository limits and collaboration permissions.
4. Inference Provider credit, provider-specific pricing, rate limits, and whether a call is billed.
5. AutoTrain availability, included credits, supported trainers, and the actual GPU price for a training run.

Training decision tree

Need better behavior?
  ├─ Better prompt/context → improve system prompt and retrieval first
  ├─ Current knowledge → add RAG with embeddings and citations
  ├─ Stable style/format → supervised LoRA pilot
  └─ New base capability → choose a larger/base model, not fine-tuning

Recommended first training pilot

• Base: a small permissively licensed instruct model, 1B–8B.
• Data: 500–5,000 high-quality examples, private and redacted.
• Method: LoRA/QLoRA via AutoTrain or a rented GPU.
• Evaluation: fixed holdout set, exact-match/JSON validity plus human review.
• Artifact: adapter, dataset revision, training config, evaluation report, and model card.

ZeroGPU should not be used as the training plan. It is valuable as the post-training inference/demo endpoint.