docs/VALIDATION.md

# FeatureLens v0.12 validation

v0.12 touches two GPU inference paths: **concept-guided discovery** (only to add split-half stability from the same activations) and the new **cross-target causal profile**. Do not rerun unchanged experiments merely as regressions.

## Local release gate

From the repository root:

```bash
python3 -m pytest -q && \
python3 -m compileall -q app.py featurelens experiments scripts && \
python3 -m ruff check app.py featurelens experiments tests scripts && \
python3 scripts/ui_smoke.py && \
python3 scripts/release_check.py
```

Expected automated test count: **65 passed**.

Expected release footer:

```text
FeatureLens release check: PASS
  discovery prompts: 224
  causal tasks: 28
  layers: [4, 14, 26]
  feature-set sizes: [1, 3, 5]
  random controls: 8
  release: v0.12.0
```

## HF acceptance — only two GPU calls

Use this Workbench context before both calls:

```text
Prompt: The derivative of x squared is
Residual layer: 14
Prompt token index: -1
```

You do not need to rerun single-feature causality, dose response, paraphrase robustness, layer trajectory, feature-set sweeps, cue diagnostics, or focus/zoom.

### GPU call 1 — touched discovery path

Exact UI path:

**Feature evidence → A. Concept-guided candidate discovery**

Set:

```text
Target concept: mathematics
Residual layer: 14
Prompts per concept: 4
Candidate features: 12
Candidate ranking: Causal-ready at current token
```

Click:

**Discover concept-associated candidates**

Pass conditions:

- The normal candidate table still appears.
- The summary includes a line beginning **Split-half shortlist stability from the same activation batch**.
- The reported Jaccard is in `[0, 1]`.
- The diagnostic is described as small-sample sensitivity, not semantic reliability.

Record only the split-half shared count/Jaccard unless the candidate ranking itself changes unexpectedly.

### GPU call 2 — new cross-target causal profile

Exact UI path:

**Feature evidence → C. Cross-target causal profile**

Set:

```text
Features for cross-target profile:
25992
16369
21670

Exact target continuations (one per line):
2x
x
0
x^2
```

Click:

**Run cross-target causal profile**

Pass conditions:

- The summary says 3 features were profiled across 4 exact continuations.
- **Cross-target causal profile** contains `3 × 4 = 12` rows.
- **Target-profile summary** contains one row per feature.
- The chart contains separate feature series across the four targets.
- No random-control/significance claim is made in this panel.

Please return:

1. the **Target-profile summary** table;
2. the 12-row **Cross-target causal profile** table (or at minimum the four rows for each feature);
3. the chart screenshot if convenient.

## Zero-GPU behavior covered by automated tests

Do not spend HF quota solely to test these:

- **Controlled evidence patterns** classifies the existing controlled specificity table without model inference.
- If **Association vs controlled causality** has no discovery table because the Space was rebuilt, it now displays an explicit state-explanation instead of staying silently blank.
- The cross-target selector prefers the target-specificity leader and JS-specificity leader when controlled results are present in the same session.

## Do not rerun for v0.12

Unless something visibly breaks, skip:

- identity paraphrase;
- layer trajectory;
- 1/3/5 feature-set sweep;
- scale dose-response;
- cue × context;
- controlled candidate specificity;
- zoom/focus behavior.

Those implementations are unchanged in v0.12 and remain covered by the automated suite.