release: add docs/REPRODUCIBILITY.md

docs/REPRODUCIBILITY.md

@@ -0,0 +1,130 @@
+# Reproducibility
+
+**Status tags:** `IMPLEMENTED` · `VERIFIED` · `MEASURED` · `NOT RUN` · `BLOCKED`.
+
+This document states exactly what a third party can reproduce, with what, and what they cannot.
+
+---
+
+## 1. The reproducibility contract
+
+| Guarantee | How it is enforced |
+|---|---|
+| **Frozen configuration** | All tunables live in `configs/base.yaml`; no magic numbers in Python. The loader validates invariants and computes a hash. |
+| **Frozen config hash** | `78f1e3700da15aa1`. Every artifact records the hash it was produced against. |
+| **Pinned backbones** | Every backbone is pinned by revision in `configs/base.yaml`; the Hub resolves the exact commit. |
+| **Seed** | `project.seed: 42`. |
+| **Immutable public test** | `evaluation.immutable_public_test: true`; `hidden_data_access: false`. |
+| **Byte-verified artifacts** | Every released artifact ships with a sha256 in `models/checksums.sha256`. |
+| **Verified metrics** | Every quoted number is checked against its artifact by `tools/verify_readme_metrics.py`. |
+
+### 1.1 Invariants the loader enforces
+
+Editing `configs/base.yaml` moves the hash and **invalidates every artifact keyed to it**. Two
+invariants exist because their violation is a *silent* error:
+
+```
+fusion.input_dim            == 3 * croma.encoder_dim + croma.optical_channels + croma.sar_channels   # 2318
+grounding_head.feature_dim  == 4 * grounding.encoder_projected_dim                                  # 2048
+```
+
+The second is especially load-bearing: a mismatch is a silent shape error that torch only raises at
+the similarity step, after patch features are already cached. It is therefore rejected at load time.
+
+## 2. Reproduce the metric verification (cheap, no GPU)
+
+```bash
+python release/tools/verify_readme_metrics.py
+```
+
+- **Reads** the artifacts under `artifacts/`.
+- **Compares** each of the 20 quoted metrics at the precision printed in the README.
+- **Also asserts** statuses (VLM headline contains `ACCEPTANCE-REJECTED`; router `corpus_limited` /
+  `n_val`; calibration temperature and `ece_improvement`).
+- **Exits 0** and prints `ALL CLAIMS VERIFIED` only when everything matches.
+
+Committed output: `release/tools/readme_metrics_report.txt`.
+
+## 3. Reproduce the environment (local run)
+
+```bash
+git clone https://github.com/Anish-lab-blip/SatQuery-AI
+cd SatQuery-AI
+python -m venv .venv
+source .venv/Scripts/activate      # Windows git-bash; .venv/bin/activate on Linux/macOS
+pip install -r requirements.txt
+```
+
+Python 3.11+ and a CPU are sufficient. Device is selected via `SATQUERY_DEVICE`; all placement is
+`.to(device)`, never `.cuda()`. Backbones are fetched from the Hugging Face Hub on first use, pinned
+by revision.
+
+## 4. Reproduce the tests
+
+| Suite | Command | Expected |
+|---|---|---|
+| Frontend live-wiring | `pytest tests/unit/test_frontend_live_wiring.py` | **106 passed** |
+| Doc/frontend suite | `pytest` on the 5 doc/frontend files | **183 passed** |
+| Full unit suite | `pytest tests/unit` | 5–6 **environmental** failures (sandbox delete guard × 4, 1 ordering flake, 1 stale adapter test) — see [`EVALUATION.md`](EVALUATION.md) §4 |
+
+> **Environment note.** In the authoring sandbox the full suite trips a **bulk-delete guard** and
+> pytest is only installed in the repository virtualenv. Neither affects a normal user environment.
+
+## 5. Reproduce a live run
+
+The deployed stack is reachable:
+
+```bash
+curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/health
+curl --noproxy '*' https://satquery-backend-m4yv.onrender.com/api/capabilities
+```
+
+`/api/capabilities` returns six tasks, all `available: true`. A live run requires the tunnel agent to
+be connected (`agent_connected:true`); if the Codespace is stopped, the request parks until the
+tunnel timeout. See [`DEPLOYMENT.md`](DEPLOYMENT.md) §5–6.
+
+## 6. Reproduce the trained artifacts
+
+Backbones are public and pinned; the **trained modules** are released on the Hugging Face Hub under
+`thundercode/SatQuery`, each with its backbone dependency and checksum. To reproduce training:
+
+| Artifact | Where it trains | Guide |
+|---|---|---|
+| router adapter | local CPU | `configs/base.yaml` §router.training |
+| grounding head | local | `configs/base.yaml` §grounding_training |
+| change head | local | `configs/base.yaml` §change |
+| optical_sar fusion head | local, seed sweep | `docs/PHASE14_OPTICAL_SAR_DECISIONS.md` |
+| change_vqa head | **external GPU (Kaggle)** | `docs/R02_KAGGLE_TRAINING_GUIDE.md` |
+| vlm LoRA adapter | **external GPU** | `configs/base.yaml` §training |
+
+### 6.1 What "reproduce" means for externally-trained artifacts
+
+The change-VQA and VLM artifacts were trained **outside this repository**. Reproducing them requires
+the external GPU environment described in their guides. The repository reproduces:
+- the **promotion gate** (byte-identity, sha256, zero non-finite tensors);
+- the **evaluation**;
+- the **serving wiring**.
+
+It does **not** ship a one-command retrain for those two artifacts. This is stated rather than
+implied.
+
+## 7. What is NOT reproducible from this release
+
+| Item | Reason |
+|---|---|
+| The private deployment repos | They are private; the deployed sources are not in this release. |
+| System-level end-to-end benchmark | **No such benchmark exists.** |
+| Router test-split number | **Not run.** |
+| CDVQA / SECOND imagery | Public but large; the release documents the acquisition + name-verification procedure, not the data. |
+| BigEarthNet full corpus | **Not downloaded** (only a 28k S2 subset was used). |
+| The historical ZeroGPU/Gradio deploy target | Frozen paperwork only; no runtime exists in code. |
+
+## 8. Environment traps recorded for reproducibility
+
+- **Dead proxy in the authoring sandbox** — outbound calls need `--noproxy '*'` (curl) or
+  `ProxyHandler({})` (Python).
+- **pytest only in the repo virtualenv** (`.venv/Scripts/python.exe`).
+- **Full-suite pytest trips a bulk-delete guard** in the sandbox.
+- **Cloudflare 308-redirects `X.html` → `/X`.**
+- **Chrome drops synthetic CDP key events when the window lacks OS focus** — relevant to any
+  browser-driven reproduction of the live validation.