docs/MASTER_SPEC.md

# MLOL — MultiDomain LLM Optimisation Lab
## Master Specification v1.1 — 2026-07-19

**Status:** approved blueprint for implementation.
**v1.1:** hardened per reliability review — positioning statement, graceful
degradation, P7/P8, config schema validation, untrusted-input handling,
runtime-aware routing, per-experiment manifests + state machine, paired
statistics, environment-stamped certificates, dependency separation, lazy
loading, mocked-backend-first delivery.
**Supersedes:** FinLLM Foundry prototype (this repo's v0; code is reused where it fits).
**Owner:** finpy1789 · Contact: finpy07@gmail.com

---

## 1. Vision

A Hugging Face Space that is a complete, research-grade **LLM fine-tuning and
optimisation laboratory**: choose a base model → upload a dataset → validate and
clean it → get a hardware recommendation → run a baseline evaluation → fine-tune
(on the right backend) → re-evaluate → compare → generate an optimisation report
and a Model Performance Certificate → publish or deploy the adapter. A
platform-wide AI Research Assistant acts as a copilot across every step.

The Space is the **control plane**. Training runs on the **training plane**
(ZeroGPU for demos, HF Jobs for serious runs, Colab export for the free route).
All state lives on the **persistence plane** (Hub repos / storage bucket).
The Space itself is stateless and restart-safe.

```
┌──────────────────────── CONTROL PLANE (this Space) ────────────────────────┐
│ upload/validation · baseline & post evals (sampled) · hardware estimation  │
│ training config · job submission · status/log monitoring · reports/certs   │
└──────────────┬─────────────────────────────────────────────┬──────────────┘
        TRAINING PLANE                                PERSISTENCE PLANE
  ├ ZeroGPU  — demo runs ≤1.5B                 ├ private dataset repo — experiments DB
  ├ HF Jobs  — managed serious training        ├ model repos — adapters/merged models
  ├ Colab export — free user-controlled        └ storage bucket — checkpoints/logs/
  └ external GPU — advanced route                 uploads (fallback: dataset repo)
```

**Positioning.** MLOL is an **orchestration and evaluation platform, not a
persistent GPU training server**. ZeroGPU provides bounded demonstrations;
large-model training is delegated to Colab, HF Jobs, or external compute.

**Graceful degradation** is an architectural objective: dataset preparation,
experiment configuration, report viewing, and Colab export must remain fully
usable when ZeroGPU is unavailable, queued, or rate-limited.

## 2. Design principles (binding)

P1. **Configuration over knowledge.** Every model, provider, domain, benchmark,
    and hardware profile lives in `configs/`. The application never validates,
    rejects, or special-cases a model name from code or from the assistant's
    internal knowledge. If it is in the config, it is valid and the UI renders
    it. Newly released models (e.g. Gemma 4, GPT-5.6) are added by editing
    config only — zero code changes. Configuration files ARE validated for
    structure at startup — required fields, types, supported capability
    combinations, and `schema_version` — via Pydantic models; a config that
    fails schema validation is reported precisely and skipped, never guessed at.
P2. **Never show an impossible button.** The routing engine decides what can run
    where; ineligible options render disabled with the reason and an alternative.
P3. **Statelessness.** Any Space restart loses nothing: all experiment state is
    on the Hub. The UI always renders from persisted state.
P4. **Statistical honesty.** Every sampled metric carries n, seed, and 95% CI.
    Certificates never imply a sample is the full benchmark.
P5. **Credential separation.** The owner token only touches demo resources and
    the platform's own experiment repo. User-owned training uses the user's
    scoped token (HF OAuth preferred). Raw tokens are never persisted or logged.
P6. **Working increments.** Each delivery increment leaves the deployed Space
    fully functional.
P7. **Hardware-aware execution.** CPU operations never acquire ZeroGPU. GPU-
    decorated functions are small, independently callable, bounded, and release
    GPU resources immediately after their operation completes.
P8. **Resumable and idempotent execution.** Every training and evaluation stage
    records its state before and after execution; retrying a completed
    operation must not create duplicate jobs or corrupt experiment artifacts.

## 3. Module map

```
MLOL
├── Home                     — pipeline overview, recent experiments, quick start
├── Tier 1 — General Fine-Tuning Lab
├── Tier 2 — Domain Foundry          (premium: access code / OAuth allowlist)
├── Evaluation Lab
├── Reports                  — optimisation reports, certificates, research dashboard
├── Adapter Library
├── Hardware Advisor
├── Documentation
└── AI Research Assistant    — docked bottom-right overlay, available on every page
```

Top-level navigation = Gradio Tabs. The assistant is a collapsible CSS-overlay
panel (bottom-right), not a tab, and is present regardless of active page.

## 4. Tier 1 — General Fine-Tuning Lab

### 4.1 Base model selection
- Dropdown fed by `configs/models.yaml`. Seed catalogue: TinyLlama, SmolLM,
  Phi-4 Mini, Qwen2.5, Qwen3, Gemma, Mistral, Llama families (editable freely).
- Each entry displays: parameters, context length, license, quantization
  support, gated flag — all from config metadata (P1).
- Schema per entry:
  ```yaml
  - name: str            # display name
    repo: str            # HF repo id
    params_b: float      # billions
    context: int
    license: str
    gated: bool
    quant: [4bit, 8bit, bf16]   # supported load modes
    chat_template: str   # LLaMA-Factory template name (for export)
    notes: str           # optional, shown verbatim
  ```

### 4.2 Dataset upload & preparation
- Accepted: CSV, JSON, JSONL, TXT, PDF, DOCX (pypdf / python-docx for extraction).
- **All preparation runs on CPU** (P7): parsing, validation, cleaning, dedupe,
  and approximate token estimation never reserve ZeroGPU. The selected model's
  tokenizer is loaded lazily and cached; a fast generic tokenizer provides
  estimates until then.
- **Untrusted input handling:** PDF/DOCX parsed as untrusted content;
  configurable limits (`configs/limits.yaml`) on upload size, extracted text
  size, sample count, and token count; encrypted, malformed, oversized, or
  decompression-heavy (zip-bomb-like) files are rejected with a clear reason.
- Pipeline: parse → column/field mapping UI (auto-guess instruction/input/output
  or prompt/response or raw text) → validate → clean (dedupe, strip empties,
  length filter, encoding fixes) → convert to chat `messages` format →
  persist to `uploaded_datasets/<run_id>/` (bucket or repo).
- Dataset Summary panel: samples, token count (tokenizer of selected model),
  average/percentile lengths, missing values, duplicates removed, language
  guess, **estimated training time per backend**.
- Validation for evaluation: if reference answers exist → reference-based
  metrics enabled; else those metrics are greyed with the reason (P2).

### 4.3 Training configuration
- **Basic mode:** epochs, batch size, learning rate, LoRA rank, alpha, dropout.
- **Advanced mode:** + scheduler, optimizer, gradient accumulation, warmup,
  weight decay, seed, method (LoRA/QLoRA/DoRA), max length, packing.
- "Recommend hyperparameters" button → rule-based suggestion from dataset size
  + model size (assistant can also do this via tool call, §9).

### 4.4 Training routing engine
```python
def choose_backend(model_params_b, sample_count, est_tokens, user_ctx) -> Route:
    # zerogpu_demo: params_b <= 1.5 and sample_count <= 5_000 and est_minutes <= window
    # hf_job:       user authenticated + jobs eligibility verified
    # colab_export: always available
    # external:     documented, config listed
```
- Routing decision panel shows: selected model, estimated GPU-hours, ZeroGPU
  eligibility (with reason), recommended backend, alternatives.
- **Routing is capability-based and runtime-aware:** eligibility (ZeroGPU
  availability, estimated duration, model memory needs, dataset size, user
  authentication, backend access) is re-checked **immediately before launch**,
  not only when the form renders. Each backend implements one
  `TrainingBackend` capability contract (also satisfied by a mock backend used
  in tests and Increment 1).
- **ZeroGPU demo path** ("Quick Demo Training — limited dataset and model size"):
  single-window LoRA run, live stage display (Preparing Dataset → Loading Base
  Model → Training → Saving Adapter → Evaluation), progress bar, loss curve
  (live-updated plot), ETA; GPU usage where the runtime exposes it.
  The demo profile is **enforced by configuration** (`configs/limits.yaml`):
  max model size, token count, sequence length, epochs, estimated duration,
  checkpoint frequency, and output size — hard limits, not UI guidance.
- **Progress without live streams:** the UI never depends on uninterrupted log
  streaming from ZeroGPU or Colab. Stage transitions and periodic summaries are
  persisted (§11), and the interface reconstructs progress from persisted state
  after any Space refresh or restart.
- **HF Jobs path:** generate config → `HfApi.run_job(...)` with user token →
  status (queued/running/completed/failed), log streaming/polling, cancel,
  adapter retrieval, auto-trigger post-eval. Eligibility check before showing.
- **Colab export path:** downloadable package
  `mlol_experiment_<run_id>/ {training_notebook.ipynb, train_config.yaml,
  dataset_manifest.json, requirements.txt, run_training.py, run_evaluation.py,
  README.md}`. The package is **pinned and self-contained**: exact dependency
  versions, dataset revision, model revision, configuration hash, resume
  instructions, and a completion-upload step. Notebook: HF auth → pull dataset
  → train (resume-capable) → push adapter → write completion metadata back to
  the experiment repo.
- **Completion verification:** a Colab/Jobs run is accepted as complete only
  after the Space verifies the expected adapter, config, and evaluation
  artifacts exist and match the original `run_id` and configuration hash;
  mismatches mark the run `failed` with the discrepancy listed.

## 5. Tier 2 — Domain Foundry (premium)

- Domains (config-driven, `configs/domains/*.yaml`): Finance, Law, Medical,
  Regulatory, Accounting, Insurance, Computer Science, Programming,
  Artificial Intelligence, Literature, General Science.
- Each domain config provides: curated dataset references, prompt/format
  templates, benchmark definitions, evaluation metric set, recommended
  hyperparameters, system prompt, disclaimer text.
- Execution: same Tier-1 machinery; domain config pre-fills everything.
  Serious runs route to HF Jobs / external; sampled evals in-Space; full evals
  via exported Job/Colab script; domain performance certificate.
- Access control: `PREMIUM_ACCESS_CODES` secret (Space-set codes) and/or HF
  OAuth username allowlist. Locked UI states name the unlock path
  (email finpy07@gmail.com).

## 6. Evaluation Lab

### 6.1 Levels
| Level    | Items    | Purpose                    | Execution                          |
|----------|----------|----------------------------|------------------------------------|
| Quick    | 25–50    | UI sanity check            | in-Space (CPU or one GPU window)   |
| Standard | 100–200  | certificate comparison     | in-Space, bounded batches (each batch a small GPU call, P7; resumable, P8) |
| Full     | complete benchmark | publication-quality | exported Job/Colab script only — never in-Space |

Per-backend execution limits (batch size, max items per window, timeouts) live
in `configs/limits.yaml`.

- Identical item IDs, seed, and generation settings for baseline vs fine-tuned
  (stored in the experiment record; re-used automatically).
- Every sampled metric reports: value, n, seed, 95% CI (Wilson for proportions,
  bootstrap for continuous). Certificates print "Full benchmark: Not executed"
  where applicable (P4).

### 6.2 Metrics
- Reference-based: accuracy (exact/normalized match), BLEU, ROUGE-L, BERTScore.
- Intrinsic: perplexity (held-out).
- Operational: latency (p50/p95), tokens/sec, peak memory, response length.
- **Hallucination — component estimate, never one opaque number:**
  factual consistency vs references · unsupported-claim rate ·
  citation/reference agreement (where applicable) · judge-model flagged rate →
  composite % labeled "Estimated hallucination risk — not a direct measurement
  of truthfulness", with judge model, prompt version, sample size, and
  human-verification flag disclosed.

### 6.3 Flow
Baseline eval (pre-training, auto) → `baseline.json` → training → identical
post eval → `post_training.json` → comparison dashboard (metric | baseline |
fine-tuned | Δ | paired significance) → classification.

- **Paired statistics:** because baseline and post-training use identical items,
  significance uses paired tests — paired bootstrap / permutation for continuous
  metrics, McNemar-style paired proportion test for accuracy-type metrics. CIs
  remain displayed on every metric (P4); CI overlap is never the significance rule.
- **Item-level persistence:** per-item outputs (or stable item hashes + scores)
  are persisted, so interrupted evaluations resume without repeating completed
  items (P8) and any comparison can be exactly reproduced later.
**Improved / Neutral / Degraded** (per-metric and overall) → Trial & Error
panel: data-driven diagnostics (dataset too small, LR too high, overfitting
signal from train/eval loss divergence, low-quality dataset signals, too few
epochs, catastrophic-forgetting probe results) with evidence for each claim.

## 7. Reports & Certificate

### 7.1 Optimisation report
Auto-generated: training summary, dataset summary, hyperparameters, evaluation
tables, performance deltas, charts (loss, metric comparisons), recommendations.
Download: **PDF** (reportlab), **CSV** (flat metrics), **JSON** (full record).

### 7.2 Model Performance Certificate (9 sections)
1. Identity: model, base, adapter, date, training time, dataset fingerprint.
2. Performance: accuracy, BLEU, BERTScore, latency, memory, hallucination
   estimate — each with n and CI.
3. Overall result: ✓ Improved / ⚠ Neutral / ✗ Degraded (CI-aware rule:
   improved = significant gains on primary metrics without significant
   regressions; degraded = any significant regression on a primary metric).
4. Confidence rating ★1–5 — reflects **evaluation comprehensiveness** (sample
   sizes, metric coverage, seed control, full-vs-sampled), never model quality.
   Rubric printed on the certificate.
5. Strengths — generated from significant positive deltas.
6. Weaknesses — generated from significant negative deltas + diagnostics.
7. Deployment recommendation: Ready / Needs More Training / Needs Better
   Dataset / Do Not Deploy (rule-based from 3+6).
8. Hardware recommendation: per-target table (Recommended / Minimum / Not
   Recommended) from `configs/hardware.yaml` profiles.
9. Research summary — templated natural-language digest of the deltas.

Every certificate (and report) embeds an **execution environment block**:
backend, accelerator (where exposed), quantization mode, model revision,
dataset fingerprint, dependency lock fingerprint (captured at runtime),
evaluation seed and sample sizes, and a demo-run vs full-run marker.

### 7.3 Research Dashboard
Experiment history (from `experiments/index.jsonl`), per-experiment graphs
(loss, BLEU, BERTScore, latency, hallucination, accuracy), compare any two
experiments side-by-side.

## 8. Adapter Library

Registry of every trained adapter: name, version, domain, base model,
performance snapshot, date, links — Download (HF repo), Deploy (§10),
Load-in-playground (inference engine hot-swap). Backed by model repos +
`evaluation_summary.json` in each.

## 9. AI Research Assistant

- **Placement:** docked bottom-right collapsible overlay, on every page.
- **Provider layer** (`configs/providers.yaml`, P1): each entry = display name,
  API type (hf-inference | openai-compatible | anthropic), model id, auth
  source (user HF OAuth token / user-supplied API key). Seed entries include
  Gemma 4, GPT-5.6, Claude, DeepSeek, Qwen — list is config, edit freely.
  User keys held in session memory only (P5).
- **Modes** (auto-selected by context + explicit switcher):
  1. **General LLM Assistant** — concept Q&A (LoRA, QLoRA, ORPO, forgetting…),
     grounded in `Documentation/` content.
  2. **Experiment Assistant** — context injection: current experiment's config,
     training logs, dataset stats, eval JSONs → data-grounded diagnosis.
  3. **Hardware Advisor** — deterministic estimator computes VRAM/time (the
     numbers), assistant explains and recommends (the words). Numbers always
     come from the estimator, not the LLM.
  4. **Report Interpreter** — certificate/report JSON injected; explains each
     metric, why the rating was assigned, highest-leverage improvements.
- **No GPU for explanations:** the assistant never acquires ZeroGPU to explain
  an experiment (P7). Default inference = HF Inference Providers or the user's
  supplied provider; local GPU inference only as an explicit option for small
  config-listed models.
- **Untrusted context:** uploaded datasets, extracted documents, logs, and
  model-generated text injected into the assistant are treated as untrusted
  data — they must not override the assistant's system instructions, and text
  inside them can never directly trigger platform actions; only the user's own
  chat turns can invoke tools.
- **Tool awareness:** function-calling against a platform action registry:
  `open_comparison(exp_a, exp_b)`, `regenerate_report(run_id)`,
  `export_certificate_pdf(run_id)`, `suggest_hyperparameters(run_id)` (pre-fills
  the training form), `navigate(module)`. Actions that mutate state are echoed
  in the chat ("Pre-filled the training form — review before launching");
  destructive actions are out of scope for the assistant.

## 10. Deployment targets

Per adapter: push to HF Hub (adapter or merged) · safetensors download ·
GGUF conversion (exported script; on-Space only for small models) · Ollama
Modelfile · Dockerfile (vLLM/TGI serving) · the Space's own inference API.
All generated artifacts, no owner-credential publishing on behalf of users (P5).

## 11. Persistence schema

```
Private dataset repo  (mlol-experiments)
  experiments/<run_id>/manifest.json          # AUTHORITATIVE per-experiment record
  experiments/<run_id>/config.yaml
  experiments/<run_id>/dataset_summary.json
  experiments/<run_id>/baseline.json
  experiments/<run_id>/post_training.json
  experiments/<run_id>/certificate.json
  experiments/<run_id>/report.pdf
  experiments/index.jsonl                     # rebuildable dashboard cache ONLY
Storage bucket (fallback: same dataset repo)
  uploaded_datasets/<run_id>/  checkpoints/<run_id>/  logs/<run_id>/  tmp/<run_id>/
Model repos (per adapter)
  adapter_config.json  adapter_model.safetensors  README.md  evaluation_summary.json
```

- **Source of truth = one manifest per experiment.** `index.jsonl` is a derived
  dashboard cache, rebuildable at any time by scanning manifests — concurrent
  runs never contend on a shared mutable file.
- **State machine** (persisted in each manifest, transitions appended with
  timestamps): `draft → data-ready → baseline-running → baseline-complete →
  training-submitted → training-running → training-complete →
  post-evaluation-running → complete`, plus `failed` and `cancelled` from any
  state. The UI renders exclusively from the recorded state (P3, P8).
- **Artifact envelope:** every persisted artifact carries `schema_version`,
  `created_at`, `updated_at`, `run_id`, config hash, model revision, dataset
  fingerprint, backend, and producer version. Small migration functions read
  older schema versions forward.
- Storage access goes through one `StorageBackend` abstraction (bucket | repo).

## 12. Tech stack

Gradio 5.x (SDK) · huggingface_hub (Jobs, repos, OAuth) · transformers/peft/trl
(demo training + inference) · datasets · evaluate + bert-score + sacrebleu ·
**pydantic** (config + experiment schema validation) · reportlab (PDF) ·
pypdf, python-docx (ingestion) · matplotlib (charts).

- **Pinned, split dependencies:** tested version ranges, not open-ended
  `latest`. `requirements.txt` = lightweight Space/control-plane deps only;
  `requirements-train.txt` = training-plane deps (used by ZeroGPU demo path,
  HF Jobs image, and Colab package). The runtime lock fingerprint is captured
  at startup for certificates (§7).
- **Service layer:** orchestration is framework-independent — Gradio callbacks
  invoke plain Python services (`src/services/`); no training, persistence, or
  routing logic lives inside UI callbacks.
- **Lazy everything:** transformers, torch, evaluation models, document
  parsers, and plotting import lazily. The Space must render with no model
  loaded and no GPU acquired (P7); first paint stays fast.

Existing FinLLM Foundry code reused: inference engine, EDGAR ingestion,
dataset mixing, training scripts (become the Jobs/Colab payload), guide logic
(absorbed into assistant mode 1).

## 13. Delivery plan (each increment ships working)

| # | Increment | Key acceptance test |
|---|-----------|--------------------|
| 1 | **Foundation**: nav skeleton, Pydantic config validation, experiment manifests + state machine, persistence recovery, `TrainingBackend` capability contract, **mocked training backend** | Space renders all modules; a mocked end-to-end run walks every state; experiments survive restart; invalid config reported precisely |
| 2a | Dataset preparation: upload→validate→clean→convert (CPU-only), limits, untrusted-input handling + **reproducible pinned Colab export** | malicious/oversized files rejected cleanly; export package runs in Colab and reports completion back |
| 2b | Bounded ZeroGPU demo training (config-enforced demo profile) | 0.5–1.5B LoRA run completes in one window; over-limit configs never reach the GPU |
| 3 | Evaluation Lab: adaptive sampling, CIs, paired tests, item-level resume, baseline/post/compare, diagnostics | same-seed before/after eval produces paired-significance dashboard; interrupted eval resumes |
| 4 | Reports + Certificate (PDF/CSV/JSON) with environment block | certificate downloads with all 9 sections + environment stamp |
| 5 | HF Jobs backend: submit, monitor, cancel, retrieve, completion verification, eligibility gating | 7B QLoRA job launched from UI with user token; tampered/mismatched artifacts rejected |
| 6 | Domain Foundry + domain benchmarks (Finance first, then the rest) | finance domain run yields domain certificate |
| 7 | AI Research Assistant (4 modes + tools, untrusted-context rules) + Adapter Library + Research Dashboard | assistant diagnoses a real experiment and pre-fills a config; injected instructions in logs are ignored |

## 14. Resolved decisions

- **Name:** MultiDomain LLM Optimisation Lab (MLOL).
- **Space:** evolve this repo/Space; rename to `finpy1789/mlol` (URL redirects).
- **Model/provider catalogues:** config-only; no code-side name validation (P1).
- **Assistant default provider:** open model via HF Inference (user OAuth);
  Claude/GPT as bring-your-own-key.
- **Premium:** access codes now; OAuth allowlist when public.