bundle/v0/HYPERGRADIENT_VALIDATION.md

# Hypergradient component validation

## Scope

This is bounded validation evidence for the analytic gradient oracle used by the independent reproduction. It is not a paper-scale experiment and does not establish either C1 or C2 by itself. C1, C2, and C3 remain `HOLD`.

The validation covers every implemented lower-level route family:

1. Gaussian portfolio with the exact Gaussian risk coefficient
2. Gaussian portfolio with the moment-class sensitivity coefficient
3. Empirical W1 portfolio
4. W1 absolute regression
5. W2 squared regression, including the root objective and post-solve square chain rule

Each route checks the distance derivative, lower-value derivative at the identity metric, lower-value derivative at the active penalty point, smoothed coverage-violation derivative, active penalty derivative, and total hypergradient. Squared regression also checks the root lower-value derivative.

## Numerical contract

- Lower-triangular central differences use steps `1e-4`, `1e-5`, and `1e-6`.
- `1e-5` is the predeclared primary step.
- A component passes only if the primary step passes and at least two of three steps pass.
- Each step passes when relative Frobenius error is at most `1e-3` or maximum absolute error is at most `1e-6`.
- Conic and transport residual thresholds are both `1e-7`.
- Every plus and minus scalar evaluation retains its solver or transport residual receipt.
- Evaluation failures are retained in the artifact failure ledger. They are never dropped from aggregation.

## Executed result

The immutable artifact was generated from clean source commit `348fb859246fd9f42bf71928ff9b654e0978c587`.

- 4 route families, 5 route variants
- 31 analytic component checks
- 93 numerical gradient matrices
- 882 retained plus/minus scalar evaluations
- 764 conic residual receipts
- 4,005 transport residual receipts
- 0 failures
- All 31 components passed all three step sizes
- Largest primary-step relative error: `7.625598602017288e-06`, empirical W1 portfolio active lower value
- Largest relative error anywhere in the step grid: `3.5048066444505585e-05`, squared-regression total hypergradient at `1e-6`
- Maximum conic residual: `3.9171789151799395e-11`
- Maximum transport residual: `3.6529009592165657e-14`

Artifact bindings:

- Artifact SHA-256: `89010cd8d96aa144f62953550cbf92061fc966ae41dd303e48ea683158632511`
- Adjacent sidecar: `89010cd8d96aa144f62953550cbf92061fc966ae41dd303e48ea683158632511  hypergradient.json`
- Evidence payload: `sha256:a822c97905c460c0b914a119195ef19e8d25834ea23d0930027194cf781215a0`
- Base Git tree: `ba5c9b5c1acad7cd56cf46678208e38f51e19695`
- Scientific files proved against Git blobs: `41`
- Source tree: `sha256:c08bf440638d1d68a3ad2a76442ab6cfd68d0aeeab2d4520d3bdde8615b4325a`
- Validation config: `sha256:a14d7dc8a9b17cd2555eeb48a91045a4ca1df0aad6533cf73836243dbf70e6e4`
- Paper plan: `sha256:f4312037cbf0a49c291ec624c8598d04507524b2d522a00be4adaacc6d716a54`
- Locked packages: NumPy `2.4.6`, SciPy `1.17.1`, CVXPY `1.9.2`, Clarabel `0.11.1`, POT `0.9.7`
- CPU: `Intel(R) Core(TM) Ultra 9 275HX`

## Reproduce and validate

From this paper lane:

```powershell
$env:OMP_NUM_THREADS="1"
$env:MKL_NUM_THREADS="1"
$env:OPENBLAS_NUM_THREADS="1"
$env:NUMEXPR_NUM_THREADS="1"
$env:PYTHONPATH="$PWD/src"
& .venv/Scripts/python.exe -B scripts/run_hypergradient_validation.py `
  --config configs/hypergradient_validation_v1.json `
  --output .openresearch/artifacts/validation/hypergradient.json

& .venv/Scripts/python.exe -B scripts/validate_hypergradient.py `
  .openresearch/artifacts/validation/hypergradient.json `
  --config configs/hypergradient_validation_v1.json
```

Create `.venv` with CPython 3.11 and install `environment/scientific-freeze.txt` exactly. `environment/requirements.lock` lists the direct scientific dependencies; the transitive freeze is the canonical regeneration environment. The runner and full validator fail closed unless NumPy, SciPy, CVXPY, Clarabel, and POT match both locks. They also force the four declared BLAS thread variables to one before importing NumPy.

The runner refuses an existing artifact or sidecar and requires a clean paper lane. It writes `hypergradient.json` plus the adjacent `hypergradient.sha256`. The validator requires that sidecar, proves every bound scientific file against the named Git commit's tree and blob objects, reconstructs each numerical gradient from retained scalar evaluations, and independently regenerates the complete evidence payload.

`--structural-only` is limited to adversarial tests. Its success message explicitly states that scientific regeneration was not run. It cannot serve as a regeneration receipt.

## Test receipt

- Hypergradient and adversarial validator suite: 12 passed
- Directly affected scientific regression suite: 78 passed
- Python compilation check: passed

Eight additional historical-canary validator tests were attempted and failed only because this isolated worktree does not contain the ignored `appendix-canary-v4` and `scientific-canary-v12` directories they copy as fixtures. They are recorded as unavailable, not as passes and not as regressions caused by this repair. The directly affected suite named above is the test evidence for this artifact.