docs/VALIDATION.md

FeatureLens v0.11 validation

v0.11 adds one new GPU inference path: Controlled candidate specificity. It does not change concept discovery, cheap candidate triage, paraphrase robustness, layer trajectory, feature-set interventions, dose response, cue diagnostics, or the validated in-place focus behavior. Do not spend ZeroGPU quota rerunning those unchanged paths.

Local release gate

Run from the repository root:

python3 -m pytest -q && \
python3 -m compileall -q app.py featurelens experiments scripts && \
python3 -m ruff check app.py featurelens experiments tests scripts && \
python3 scripts/ui_smoke.py && \
python3 scripts/release_check.py

Expected release checker tail:

FeatureLens release check: PASS
  discovery prompts: 224
  causal tasks: 28
  layers: [4, 14, 26]
  feature-set sizes: [1, 3, 5]
  random controls: 8
  release: v0.11.0

HF acceptance: one GPU call only

The v0.10 discovery and triage paths were already validated. For v0.11, test only the new controlled follow-up.

Exact path

Open Feature evidence and scroll to:

B. Batched causal candidate triage → Controlled candidate specificity

Use the current Workbench context:

Prompt: The derivative of x squared is
Residual layer: 14
Prompt token index: -1

In Candidates for controlled comparison, manually enter/select exactly:

16369
25992
21670

These are the three strategic candidates established by the validated v0.10 workflow:

• 16369: discovery leader and next-token-JS leader;
• 25992: raw target-effect leader;
• 21670: second-largest raw target effect.

Set:

Controlled target continuation: 2x

Click Run controlled candidate comparison.

Pass conditions

The run should produce Controlled candidate specificity with exactly three feature rows. For every row verify:

• Active at current token is True;
• Target specificity ratio is finite and non-negative;
• Target empirical tail p lies in (0, 1];
• JS specificity ratio is finite and non-negative;
• JS empirical tail p lies in (0, 1];
• random-control means/stds are populated;
• the summary says each candidate used 8 norm-matched random controls;
• execution-context null drift is reported separately from the causal effect.

The plot Random-normalized causal specificity should show two series per feature:

Target specificity
JS specificity

Do not assume which feature must win. That is the empirical result this version is intended to measure.

Optional integration check — no additional GPU call

If the Space still has the v0.10 discovery table populated in the same browser session, the Association vs controlled causality section should also populate automatically.

Verify that Discovery–controlled-causality alignment contains the same controlled features and includes:

• discovery rank;
• specificity rank;
• target-effect rank;
• JS-specificity rank;
• target specificity ratio;
• target empirical tail p;
• JS specificity ratio;
• JS empirical tail p;
• discovery→specificity rank shift.

The summary may include descriptive Spearman correlations. With only three controlled candidates, these are diagnostic summaries, not significance claims.

If the discovery table is not populated because the Space was freshly rebuilt, do not spend two extra GPU calls merely to repopulate it for v0.11 acceptance. The zero-GPU join logic is covered by automated tests.

Do not rerun for v0.11

Do not rerun these paths unless you independently notice a regression:

• Concept-guided candidate discovery;
• Batched causal candidate triage;
• Scale dose-response;
• Single-feature causal test;
• Paraphrase identity control;
• Layer trajectory;
• 1/3/5 feature-set sweep;
• cue sensitivity or cue × context;
• plot/table focus behavior.

Those implementations were not changed in v0.11.

What to send back

For the new controlled comparison, send:

1. the three feature rows from Controlled candidate specificity;
2. the summary above the table;
3. optionally the Association vs controlled causality summary/table if it populated without extra GPU work.

That is sufficient to decide whether the live feature set should now be frozen and the project moved to the full offline benchmark.