docs/USER_GUIDE.md

MLOL User Guide

MultiDomain LLM Optimisation Lab — fine-tune, evaluate, and certify open LLMs from one control panel. Full design: docs/MASTER_SPEC.md.

The pipeline

1. Tier 1 → Prepare dataset. Pick a base model, upload CSV/JSON/JSONL/TXT/ PDF/DOCX. MLOL validates, cleans, deduplicates, converts to chat format, and creates an experiment (everything is persisted; restarts lose nothing).
2. Check routing. MLOL tells you where this run can execute:
   • ZeroGPU demo — models ≤1.5B, ≤5k samples, minutes-long runs, on this Space.
   • Colab export — a pinned, self-contained package; free GPU, your account.
   • HF Jobs — managed training on paid HF hardware (needs your token/billing). Impossible options are disabled with the reason.
3. Baseline eval — before training, on a seeded sample with fixed item IDs.
4. Fine-tune — LoRA on the routed backend.
5. Post-eval + compare — identical items/seed; paired permutation tests decide significance; the Trial & Error panel explains disappointing results with evidence (dataset too small, overfitting, forgetting…).
6. Reports tab — optimisation report + 9-section Model Performance Certificate (PDF/CSV/JSON) with a full environment stamp.

Tier 2 — Domain Foundry (premium)

Pre-configured domains (Finance, Law, Medical, Regulatory, Programming, …) with curated datasets, benchmarks, and recommended hyperparameters. Access codes: email finpy07@gmail.com.

AI Research Assistant (bottom-right)

Four modes: General (concepts), Experiment (diagnoses your run from its actual logs/metrics), Hardware (estimator-grounded advice), Report (explains your certificate). Providers are configurable; Claude/GPT need your own API key (never stored).

Honest-numbers policy

Every sampled metric shows n, seed, and a 95% CI. Certificates state that sampled results are not the full benchmark, and hallucination figures are labeled estimates. Confidence stars rate how thorough the evaluation was — never how good the model is.

Interpreting states

draft → data-ready → baseline-running → baseline-complete → training-submitted → training-running → training-complete → post-evaluation-running → complete (plus failed/cancelled, restartable). The dashboard reconstructs progress from persisted state after any refresh.