docs/VALIDATION.md

FeatureLens v0.7 validation

This guide tests the v0.7 changes plus a small number of high-value regressions. Use the exact UI labels below. The comprehensive edge/adversarial suite remains deferred until final hardening.

A. Local release gates

From the repository root:

python3 -m pytest -q
python3 -m compileall -q app.py featurelens experiments scripts
python3 scripts/ui_smoke.py
python3 scripts/release_check.py

Expected:

45 passed
FeatureLens UI launch smoke: PASS
FeatureLens release check: PASS
  discovery prompts: 224
  causal tasks: 28
  layers: [4, 14, 26]
  feature-set sizes: [1, 3, 5]
  random controls: 8
  release: v0.7.0

B. UI cleanup, table typography, focus mode, and export names

1. Open Workbench.
2. Confirm these are normal always-visible sections, not collapsible dropdowns/accordions:
   • III. Single-feature scale dose-response
   • IV. Contrastive continuation preference
3. Confirm there is only one experiment heading before each section. A plot title such as Scale dose-response is fine; there should not be a second accordion heading with the same text.
4. After any result table appears, confirm its table title (for example Strongest active SAE features) is visually larger/bolder than ordinary body text, and column headings are clearly readable.
5. Produce a plot and click its top-right fullscreen/focus icon.
   • The focused plot should open near the top-center of the viewport.
   • It should be bounded rather than stretching across the entire monitor.
   • Axis labels should remain readable without needing to pan horizontally across a huge plot.
6. Exit focus mode. You should return to approximately the same page location you were at before opening it.
7. Repeat the focus test on one wide result table.
8. Click the plot export icon. Verify the downloaded filename is descriptive, for example:

featurelens_activation-profile.png
featurelens_scale-dose-response.png

It should not be the generic chart.png. 9. Click Copy table with headers and confirm the temporary ✓ Copied with headers feedback still appears.

C. Establish the shared Workbench context

In Workbench → I. Inspect a prompt location set:

Prompt: The derivative of x squared is
Residual layer: 14
Prompt token index: -1
Displayed active features: 12

Click Inspect sparse features.

Expected regression values are approximately:

selected token: 5 (' is')
feature 22632 activation: 32.90625
reconstruction cosine: 0.9015
NMSE: 0.1874
Top-5 activation mass: 26.7%

The persistent Current Workbench context banner must state the inspected prompt, layer 14, and token 5. It should remain visible when switching tabs.

D. German-language control regression

Open Feature evidence → A. Concept-guided candidate discovery and inspect Target concept.

Pass conditions:

• german_language is available;
• french_language is not available.

The controlled examples in Workbench should also include a German example such as:

Ich möchte einen Tisch für zwei reservieren.

No GPU run is required for this check.

E. Balanced concept-guided candidate discovery

Open Feature evidence → A. Concept-guided candidate discovery.

Set:

Target concept: mathematics
Residual layer: 14
Prompts per concept: 4
Candidate features: 12
Candidate ranking: Balanced selectivity

Keep the Workbench context from Test C and click Discover concept-associated candidates.

Verify that Candidate feature evidence includes these fields:

Rank
Feature id
Candidate score
Target mean max
Other mean max
Mean difference
Selectivity
Target activation rate
Other activation rate
Current prompt max
Current token activation
Active at current token

Interpretation check:

• Candidate score is an exploratory balanced score, not a semantic label.
• Globally huge but weakly selective features should no longer automatically dominate merely because their raw coefficients are enormous.
• Current prompt max and Current token activation show whether a candidate can immediately be tested in the current Workbench context.
• Selected candidate feature id should default to the highest-ranked displayed candidate active at the current Workbench token when such a candidate exists; otherwise it may default to the top-ranked candidate.

Record the top 12 rows. These are the most scientifically useful new v0.7 results.

E2. Raw-ranking comparison

Without changing the other fields, set:

Candidate ranking: Raw mean difference

and rerun Discover concept-associated candidates.

This should reproduce the scale-sensitive style of ranking used previously. The purpose of this test is to verify that v0.7 exposes the methodological choice explicitly instead of silently changing it.

You do not need to investigate all 12 raw-ranking features causally.

F. Candidate row selection and explicit reuse

After Test E:

1. Click a row in Candidate feature evidence. The row's feature id should appear in Selected candidate feature id.
2. Click Use selected candidate across feature tests.
3. A visible confirmation should say that the feature was loaded.
4. Verify the same id appears in:
   • Workbench → Single feature id;
   • Workbench → Dose-response feature id;
   • Workbench → Contrastive feature id;
   • Feature evidence → Feature id.

This is a state-routing operation and should not consume another GPU call.

G. Completion-cue regression for feature 22632

Open Feature evidence → C. Completion-cue sensitivity.

Set:

Feature id: 22632
Residual layer: 14
Prompt stem: The derivative of x squared
Completion cues:
is
=
:
equals
therefore

Click Run completion-cue scan.

The is condition reconstructs the original prompt and should remain approximately:

activation: 32.90625
Active in TopK: True

The previous reference result had zero activation for =, :, equals, and therefore. Treat that as a regression reference, not as a hard scientific requirement if tiny numerical differences appear.

H. Cue × context specificity

Open Feature evidence → D. Cue × context specificity.

Use:

Feature id: 22632
Residual layer: 14

Prompt stems:
The derivative of x squared
The capital of Germany
The weather today
My name

Completion cues:
is
=
:
equals
therefore

Click Run cue × context scan.

Expected output:

• 20 rows = 4 stems × 5 cues;
• columns Prompt stem, Cue, Full prompt, Final token, Activation, Active in TopK;
• a Cue response across contexts plot.

Scientific question: does feature 22632 fire specifically on the literal is, on is across many contexts, on several completion-boundary cues, or only in the original mathematics context?

Send the active rows and the per-cue context summary. This is the other key new v0.7 result.

I. Prompt-wide selected-feature contrast with German control

Still in Feature evidence, set:

Feature id: 22632
Residual layer: 14
Prompts per concept: 4

Click Run controlled concept contrast.

The result should contain german_language rather than french_language while preserving seven balanced concepts and 28 sampled prompts total.

Do not expect the exact old French-based concept means to remain identical because one control group has intentionally changed.

J. High-value existing regressions

Only confirm these run; do not resend full tables unless something changes unexpectedly.

1. Workbench → III. Single-feature scale dose-response with feature 22632, target 2x:
   • multiplier 1.0 must remain the exact same-batch numerical null with all causal deltas equal to zero.
2. Paraphrase robustness default pair:
   • selected-token cosine should remain around 0.301;
   • prompt-wide cosine should remain around 0.981.
3. Paraphrase robustness identical-prompt control:
   • all four robustness metrics should remain 1.000.
4. Layer trajectory:
   • layer 14 should remain the weakest reconstruction of layers 4/14/26 on the default prompt.
5. Feature sets → Set-size sensitivity → Run 1/3/5-feature ablation sweep:
   • this is the top-k causal sweep;
   • it is not the Workbench Strongest active SAE features table;
   • confirm k = 1, 3, 5 rows appear.
6. After dynamic results expand, scroll to the actual bottom without switching tabs. The v0.5 reflow fix should remain intact.

K. Deferred final hardening

Do not spend ZeroGPU quota on the complete adversarial suite yet. Preserve these for the final hardening release:

• empty prompts;
• out-of-range token indices;
• feature ids 0, 32767, invalid 32768;
• inactive feature ablation vs injection;
• whitespace-sensitive targets such as 2x vs 2x;
• missing required target continuations;
• unrelated paraphrases;
• negation and mixed-language prompts;
• long-prompt truncation;
• rapid repeated clicks / queue behavior;
• narrow/mobile-width layout;
• final copy/focus/export sweep.