docs/HF_PRO_VALUE_PLAN.md

# HF PRO Value Plan

## What we know from this project

| Capability | Current evidence | Intended use |
|---|---|---|
| Private Space | `abalanescu/flow` is private and deployed | Host the private provider UI/API |
| ZeroGPU (PRO) | Gemma 26B and Qwen 27B Q6 returned live responses | Multi-model private OpenAI upstream |
| Bucket mount | GGUF files are mounted under `/data` | Keep 23GB+ model weights persistent across restarts |
| Dynamic Hardware | A10G (24GB) / RTX Pro 6000 (48GB `large`) verified | Run quantized 26B–27B models on-demand |
| Inference Providers | Previous FLUX/Whisper calls returned 402 | Treat credit as scarce, optional spend |

## Verified ZeroGPU Architecture & Sizing Rules

1. **Hardware & VRAM**:
   - `zero-a10g` assigns 24GB VRAM (A10G) by default; new ZeroGPU infrastructure supports `large` (48GB) and `xlarge` (96GB, 2x quota cost).
   - Model weights vs KV-cache: For a ~22.9 GB model (Qwen 27B Q6_K) on 24GB VRAM, remaining VRAM is ~1.1 GB.
   - Context sizing: `n_ctx=8192` safely fits KV-cache in memory. `n_ctx=0` attempts to pre-allocate full 262k context (~16GB KV cache), which triggers CUDA OOM during context initialization.

2. **ZeroGPU Supervisor Integration**:
   - Handlers must be registered through Gradio's event graph at `demo.launch(prevent_thread_lock=True, ssr_mode=False)`.
   - OpenAI routes (`/v1/models`, `/v1/chat/completions`) mount onto `demo.app.add_api_route(...)` to share the single listener on port 7860 without conflicting with Space supervisor probes.

3. **Persistent Bucket Storage**:
   - GGUF weights stored in `/data` avoid cold-start download latency.
   - Uncached models use `hf_hub_download` into `/data` on first request.

## What must be measured, not assumed

HF plan benefits and quotas change. Verify them in the account UI and official docs before budgeting:

1. ZeroGPU daily quota, queue priority, duration limits, hardware actually assigned, and whether quota is per-user or per-Space.
2. Private Space count, hardware eligibility, secrets, storage, buckets, and bandwidth.
3. Private model and dataset repository limits and collaboration permissions.
4. Inference Provider credit, provider-specific pricing, rate limits, and whether a call is billed.
5. AutoTrain availability, included credits, supported trainers, and the actual GPU price for a training run.

## Training decision tree

```text
Need better behavior?
  ├─ Better prompt/context → improve system prompt and retrieval first
  ├─ Current knowledge → add RAG with embeddings and citations
  ├─ Stable style/format → supervised LoRA pilot
  └─ New base capability → choose a larger/base model, not fine-tuning
```

### Recommended first training pilot

- Base: a small permissively licensed instruct model, 1B–8B.
- Data: 500–5,000 high-quality examples, private and redacted.
- Method: LoRA/QLoRA via AutoTrain or a rented GPU.
- Evaluation: fixed holdout set, exact-match/JSON validity plus human review.
- Artifact: adapter, dataset revision, training config, evaluation report, and model card.

ZeroGPU should not be used as the training plan. It is valuable as the post-training inference/demo endpoint.