docs/VALIDATION.md

FeatureLens v0.12 validation

v0.12 touches two GPU inference paths: concept-guided discovery (only to add split-half stability from the same activations) and the new cross-target causal profile. Do not rerun unchanged experiments merely as regressions.

Local release gate

From the repository root:

python3 -m pytest -q && \
python3 -m compileall -q app.py featurelens experiments scripts && \
python3 -m ruff check app.py featurelens experiments tests scripts && \
python3 scripts/ui_smoke.py && \
python3 scripts/release_check.py

Expected automated test count: 65 passed.

Expected release footer:

FeatureLens release check: PASS
  discovery prompts: 224
  causal tasks: 28
  layers: [4, 14, 26]
  feature-set sizes: [1, 3, 5]
  random controls: 8
  release: v0.12.0

HF acceptance — only two GPU calls

Use this Workbench context before both calls:

Prompt: The derivative of x squared is
Residual layer: 14
Prompt token index: -1

You do not need to rerun single-feature causality, dose response, paraphrase robustness, layer trajectory, feature-set sweeps, cue diagnostics, or focus/zoom.

GPU call 1 — touched discovery path

Exact UI path:

Feature evidence → A. Concept-guided candidate discovery

Set:

Target concept: mathematics
Residual layer: 14
Prompts per concept: 4
Candidate features: 12
Candidate ranking: Causal-ready at current token

Click:

Discover concept-associated candidates

Pass conditions:

• The normal candidate table still appears.
• The summary includes a line beginning Split-half shortlist stability from the same activation batch.
• The reported Jaccard is in [0, 1].
• The diagnostic is described as small-sample sensitivity, not semantic reliability.

Record only the split-half shared count/Jaccard unless the candidate ranking itself changes unexpectedly.

GPU call 2 — new cross-target causal profile

Exact UI path:

Feature evidence → C. Cross-target causal profile

Set:

Features for cross-target profile:
25992
16369
21670

Exact target continuations (one per line):
2x
x
0
x^2

Click:

Run cross-target causal profile

Pass conditions:

• The summary says 3 features were profiled across 4 exact continuations.
• Cross-target causal profile contains 3 × 4 = 12 rows.
• Target-profile summary contains one row per feature.
• The chart contains separate feature series across the four targets.
• No random-control/significance claim is made in this panel.

Please return:

1. the Target-profile summary table;
2. the 12-row Cross-target causal profile table (or at minimum the four rows for each feature);
3. the chart screenshot if convenient.

Zero-GPU behavior covered by automated tests

Do not spend HF quota solely to test these:

• Controlled evidence patterns classifies the existing controlled specificity table without model inference.
• If Association vs controlled causality has no discovery table because the Space was rebuilt, it now displays an explicit state-explanation instead of staying silently blank.
• The cross-target selector prefers the target-specificity leader and JS-specificity leader when controlled results are present in the same session.

Do not rerun for v0.12

Unless something visibly breaks, skip:

• identity paraphrase;
• layer trajectory;
• 1/3/5 feature-set sweep;
• scale dose-response;
• cue × context;
• controlled candidate specificity;
• zoom/focus behavior.

Those implementations are unchanged in v0.12 and remain covered by the automated suite.