docs/VALIDATION.md

FeatureLens v0.10 validation

v0.10 does not change paraphrase robustness, layer trajectory, feature-set interventions, dose-response inference, cue × context, or the in-place focus implementation. Do not spend ZeroGPU quota rerunning those paths.

The only new live behavior is the association-to-causality synthesis computed after the existing discovery + candidate-triage workflow.

1. Local release gate — no HF GPU

From the repository root:

python3 -m pytest -q && \
python3 -m compileall -q app.py featurelens experiments scripts && \
python3 -m ruff check app.py featurelens experiments tests scripts && \
python3 scripts/ui_smoke.py && \
python3 scripts/release_check.py

Expected high-level result:

• all tests pass;
• Ruff reports All checks passed!;
• FeatureLens UI launch smoke: PASS;
• release check ends with release: v0.10.0.

Stop if any local gate fails.

---

HF acceptance — two GPU calls total

Use the existing Workbench context:

• Workbench → Prompt: The derivative of x squared is
• Workbench → Residual layer: 14
• Workbench → Prompt token index: -1

You do not need to click Inspect sparse features first.

2. GPU call 1 — causal-ready mathematics discovery

Exact path:

Feature evidence → A. Concept-guided candidate discovery

Set exactly:

• Target concept: mathematics
• Residual layer: 14
• Prompts per concept: 4
• Candidate features: 12
• Candidate ranking: Causal-ready at current token

Click:

Discover concept-associated candidates

Expected regression behavior from v0.9:

• the summary should say 12/12 displayed candidates are active at the selected Workbench token;
• Candidate features to screen should auto-populate with the first five returned feature IDs;
• no additional UI/focus regression testing is required because that code was not changed in v0.10.

For the current canonical prompt, the v0.9 result began with candidates 16369, 5712, 26112, 25992, 21670. Exact floating-point values can vary slightly, but a major ordering change should be reported.

3. GPU call 2 — triage + association-to-causality synthesis

Exact path:

Feature evidence → B. Batched causal candidate triage

Leave the five auto-populated candidates selected.

Set:

• Screen target continuation: 2x

Click:

Screen candidate ablations

The normal Candidate ablation screen should appear first.

Immediately below it, without another GPU action, v0.10 should populate:

Association vs causal influence

with:

1. a descriptive summary;
2. Discovery–causality alignment table;
3. Association evidence vs target effect scatter plot.

Required alignment-table columns

• Feature id
• Discovery rank
• Target-effect rank
• Distribution-shift rank
• Candidate score
• Selectivity
• Current token activation
• |Δ mean log p/token|
• Next-token JS
• Discovery→target rank shift

Required summary behavior

The summary must identify separately:

• top discovery candidate;
• strongest target-effect candidate;
• strongest next-token distribution-shift candidate;
• Spearman ρ(candidate score, |target effect|);
• Spearman ρ(candidate score, next-token JS);
• an explicit warning that the live correlations are descriptive because the screened set is small and triage has no random-control ensemble.

For the exact v0.9 values you reported, the expected qualitative pattern is:

• discovery rank #1: feature 16369;
• strongest target effect: feature 25992;
• strongest next-token JS shift: feature 16369;
• discovery score versus target-effect magnitude: strongly negative descriptive rank correlation;
• discovery score versus JS: positive but weaker descriptive rank correlation.

Do not require exact decimals as a pass condition.

Rank-shift sanity check

For the v0.9 ordering:

• feature 25992: discovery rank 4, target-effect rank 1 → Discovery→target rank shift = +3;
• feature 16369: discovery rank 1, target-effect rank 5 → rank shift -4.

This is the most important v0.10 regression check because it demonstrates that concept-evidence rank and target-causal rank are not interchangeable.

---

4. What to send back

Only send:

1. the v0.10 Candidate ablation screen if it changed materially from v0.9;
2. the new Discovery–causality alignment table;
3. the new association/causality summary with the two Spearman values;
4. optionally a screenshot of Association evidence vs target effect if the plot looks wrong.

Do not rerun identity paraphrase, layer trajectory, 1/3/5 sweep, dose response, cue × context, or focus behavior for v0.10. Those implementations were not changed.

---

Final hardening later

The broad adversarial/release suite remains deferred until the live feature set is frozen. After v0.10 acceptance, the next high-value step should be the real offline held-out benchmark rather than another round of unrelated live widgets.