docs/VALIDATION.md

# FeatureLens v0.10 validation

v0.10 does **not** change paraphrase robustness, layer trajectory, feature-set interventions, dose-response inference, cue × context, or the in-place focus implementation. Do not spend ZeroGPU quota rerunning those paths.

The only new live behavior is the **association-to-causality synthesis** computed after the existing discovery + candidate-triage workflow.

## 1. Local release gate — no HF GPU

From the repository root:

```bash
python3 -m pytest -q && \
python3 -m compileall -q app.py featurelens experiments scripts && \
python3 -m ruff check app.py featurelens experiments tests scripts && \
python3 scripts/ui_smoke.py && \
python3 scripts/release_check.py
```

Expected high-level result:

- all tests pass;
- Ruff reports `All checks passed!`;
- `FeatureLens UI launch smoke: PASS`;
- release check ends with `release: v0.10.0`.

Stop if any local gate fails.

---

# HF acceptance — two GPU calls total

Use the existing Workbench context:

- **Workbench → Prompt:** `The derivative of x squared is`
- **Workbench → Residual layer:** `14`
- **Workbench → Prompt token index:** `-1`

You do **not** need to click **Inspect sparse features** first.

## 2. GPU call 1 — causal-ready mathematics discovery

Exact path:

**Feature evidence → A. Concept-guided candidate discovery**

Set exactly:

- **Target concept:** `mathematics`
- **Residual layer:** `14`
- **Prompts per concept:** `4`
- **Candidate features:** `12`
- **Candidate ranking:** `Causal-ready at current token`

Click:

**Discover concept-associated candidates**

Expected regression behavior from v0.9:

- the summary should say `12/12` displayed candidates are active at the selected Workbench token;
- **Candidate features to screen** should auto-populate with the first five returned feature IDs;
- no additional UI/focus regression testing is required because that code was not changed in v0.10.

For the current canonical prompt, the v0.9 result began with candidates `16369`, `5712`, `26112`, `25992`, `21670`. Exact floating-point values can vary slightly, but a major ordering change should be reported.

## 3. GPU call 2 — triage + association-to-causality synthesis

Exact path:

**Feature evidence → B. Batched causal candidate triage**

Leave the five auto-populated candidates selected.

Set:

- **Screen target continuation:** `2x`

Click:

**Screen candidate ablations**

The normal **Candidate ablation screen** should appear first.

Immediately below it, without another GPU action, v0.10 should populate:

### `Association vs causal influence`

with:

1. a descriptive summary;
2. **Discovery–causality alignment** table;
3. **Association evidence vs target effect** scatter plot.

### Required alignment-table columns

- `Feature id`
- `Discovery rank`
- `Target-effect rank`
- `Distribution-shift rank`
- `Candidate score`
- `Selectivity`
- `Current token activation`
- `|Δ mean log p/token|`
- `Next-token JS`
- `Discovery→target rank shift`

### Required summary behavior

The summary must identify separately:

- top discovery candidate;
- strongest target-effect candidate;
- strongest next-token distribution-shift candidate;
- Spearman `ρ(candidate score, |target effect|)`;
- Spearman `ρ(candidate score, next-token JS)`;
- an explicit warning that the live correlations are descriptive because the screened set is small and triage has no random-control ensemble.

For the exact v0.9 values you reported, the expected qualitative pattern is:

- discovery rank #1: feature `16369`;
- strongest target effect: feature `25992`;
- strongest next-token JS shift: feature `16369`;
- discovery score versus target-effect magnitude: strongly negative descriptive rank correlation;
- discovery score versus JS: positive but weaker descriptive rank correlation.

Do not require exact decimals as a pass condition.

### Rank-shift sanity check

For the v0.9 ordering:

- feature `25992`: discovery rank `4`, target-effect rank `1` → `Discovery→target rank shift = +3`;
- feature `16369`: discovery rank `1`, target-effect rank `5` → rank shift `-4`.

This is the most important v0.10 regression check because it demonstrates that concept-evidence rank and target-causal rank are not interchangeable.

---

## 4. What to send back

Only send:

1. the v0.10 **Candidate ablation screen** if it changed materially from v0.9;
2. the new **Discovery–causality alignment** table;
3. the new association/causality summary with the two Spearman values;
4. optionally a screenshot of **Association evidence vs target effect** if the plot looks wrong.

Do **not** rerun identity paraphrase, layer trajectory, 1/3/5 sweep, dose response, cue × context, or focus behavior for v0.10. Those implementations were not changed.

---

# Final hardening later

The broad adversarial/release suite remains deferred until the live feature set is frozen. After v0.10 acceptance, the next high-value step should be the real offline held-out benchmark rather than another round of unrelated live widgets.