ArchitSharma commited on
Commit
9838759
·
1 Parent(s): 3a2b2e4

Release FeatureLens v0.15.0

Browse files
.gitignore CHANGED
@@ -11,4 +11,4 @@ artifacts/*.npz
11
  *.pt
12
  *.safetensors
13
  pycache/
14
- venv/
 
11
  *.pt
12
  *.safetensors
13
  pycache/
14
+ artifacts/*.complete
CHANGELOG.md CHANGED
@@ -1,5 +1,17 @@
1
  # Changelog
2
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ## v0.14.0
4
 
5
  - Transitioned the project from live-feature expansion toward the full offline empirical study.
 
1
  # Changelog
2
 
3
+ ## v0.15.0
4
+
5
+ - Reworked the public Gradio surface around a documented **research-instrument design system** rather than SaaS/dashboard defaults.
6
+ - Added `DESIGN.md` with typography, color, spacing, surface, button, table, plot, and anti-pattern rules so future UI edits have explicit constraints.
7
+ - Replaced the three-column onboarding/card pattern with compact editorial guidance; flattened repeated context/callout treatment; removed visible release marketing; shortened result copy so measured values lead and methodology lives in the Method tab/docs.
8
+ - Introduced a two-typeface hierarchy (serif display headings, neutral sans-serif controls/data), tighter semantic spacing, compact primary actions, quiet utility buttons, and stronger explicit table headings.
9
+ - Normalized the dynamic cross-target plots to a restrained three-series palette instead of Vega's saturated categorical defaults.
10
+ - Rewrote the public README around the research question, live tool, offline study, and reproducible workflow instead of a long release-history narrative.
11
+ - Added a ready-to-run Google Colab notebook plus `docs/COLAB.md` for Drive-backed artifact persistence and resumable study execution.
12
+ - Added `--activation-batch-size` / `--activation-max-length` to the full runner and task-level checkpoint/resume support inside causal and feature-set stages.
13
+ - Added automated design-contract regression tests.
14
+
15
  ## v0.14.0
16
 
17
  - Transitioned the project from live-feature expansion toward the full offline empirical study.
DESIGN.md ADDED
@@ -0,0 +1,112 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # FeatureLens design system
2
+
3
+ FeatureLens is a research instrument. Its interface should help someone **operate** an experiment and **read** evidence; it is not a landing page, growth dashboard, or software-company product surface.
4
+
5
+ ## Visual direction
6
+
7
+ The visual reference is a compact scientific workbench: editorial headings, neutral controls, dense but readable tables, restrained plots, and very little decorative chrome. The interface should feel designed by someone who expects to inspect numbers for a while.
8
+
9
+ ### Typography
10
+
11
+ - Display / section headings: `Georgia`, `Cambria`, `Times New Roman`, serif fallback.
12
+ - Body, controls, table cells: `Segoe UI`, `Helvetica Neue`, `Arial`, sans-serif fallback.
13
+ - Monospace is reserved for token chips and literal identifiers where fixed-width text helps scanning.
14
+ - Hierarchy comes from scale, weight, and spacing—not from badges, italics, all-caps kickers, or a different treatment for every subsection.
15
+ - Long explanatory prose should stay near 70–80 characters per line even though the analytical canvas is wide.
16
+
17
+ ### Color roles
18
+
19
+ - Accent: muted teal `#6F8984`.
20
+ - Secondary series: umber `#8A735D`, red-grey `#8C6A67`, plum `#786F82`, stone `#82827E`, blue-grey `#687982`.
21
+ - Neutrals come from Gradio's grey theme tokens so light and dark appearance remain usable.
22
+ - Saturated default categorical chart colors should be avoided when a fixed series set is known.
23
+ - No gradients, neon accents, glows, glass effects, or decorative beige/cream surfaces.
24
+
25
+ ### Geometry and surfaces
26
+
27
+ - Radius: approximately 2 px. Controls may be slightly rounded but should never look pill-shaped.
28
+ - No nested-card hierarchy. Groups exist for layout/state only and are visually flat.
29
+ - Avoid drop shadows. The only shadow permitted is the temporary in-place focus mode for dense tables/plots.
30
+ - Use rules, alignment, and whitespace instead of containers-within-containers.
31
+ - Do not add left-border "accent cards" or side-tab callouts.
32
+
33
+ ### Spacing
34
+
35
+ - Spacing is intentionally uneven by semantic role: related controls are tight; a new experiment has more separation.
36
+ - Section headings should sit close to the controls/results they introduce.
37
+ - Table titles use the otherwise-empty table toolbar band rather than consuming another row of vertical space.
38
+ - Do not introduce uniform 16/24/32 px spacing everywhere simply because it is convenient.
39
+
40
+ ## Components
41
+
42
+ ### Header
43
+
44
+ One title and one factual subtitle. Do not show version badges, status chips, marketing claims, or repository trivia in the live header.
45
+
46
+ ### Tabs
47
+
48
+ Flat text navigation with one active underline. No tab pills or raised nav cards.
49
+
50
+ ### Buttons
51
+
52
+ - Primary experiment action: compact muted-teal button.
53
+ - Utility action (`Copy TSV`, candidate hand-off): quiet secondary button.
54
+ - Button text should describe the action (`Rank candidates`, `Run intervention`) rather than generic calls to action (`Continue`, `Get started`).
55
+ - Do not make every button full-width on desktop.
56
+
57
+ ### Forms
58
+
59
+ Labels are literal and stable because validation instructions refer to them. Helper text is only included when the field's semantics are not obvious from the label.
60
+
61
+ ### Tables
62
+
63
+ Tables are primary research objects, not decorative cards. Use explicit headings, normal-weight data, tabular numerals, bounded height, horizontal scrolling for genuinely wide schemas, and a quiet `Copy TSV` utility.
64
+
65
+ ### Plots
66
+
67
+ Plots use a small, restrained palette and preserve their aspect ratio in focus mode. A plot should not expand to fill every available screen dimension. Export filenames should describe the figure rather than defaulting to `chart.png`.
68
+
69
+ ### Results copy
70
+
71
+ Measured values first. Caveats should be local and short; long methodological qualification belongs in **Method** or the offline report. Avoid paragraphs that restate what the immediately adjacent table already shows.
72
+
73
+ ## Information architecture
74
+
75
+ - **Guide** — how the evidence ladder works.
76
+ - **Workbench** — inspect one residual location and intervene on one feature.
77
+ - **Feature sets** — joint-feature interventions and geometry.
78
+ - **Features** — discovery, triage, controlled comparison, target profiles, and feature diagnostics.
79
+ - **Paraphrases** — robustness to rewording.
80
+ - **Layers** — early/middle/late SAE trajectory.
81
+ - **Study** — artifact-backed offline results only.
82
+ - **Method** — formulas, control discipline, and interpretation boundaries.
83
+
84
+ The global context line is the single source of truth for the current Workbench prompt/layer/token. Do not repeat separate context cards in each tab.
85
+
86
+ ## Anti-patterns
87
+
88
+ Do not add any of the following without a specific functional reason:
89
+
90
+ - card grids for simple prose;
91
+ - cards nested inside cards;
92
+ - status-chip or badge collections;
93
+ - gradients, glassmorphism, glow, or oversized decorative backgrounds;
94
+ - AI/SaaS language such as "unlock", "supercharge", "powerful", or "all-in-one";
95
+ - eyebrow/kicker labels above ordinary headings;
96
+ - tiny numbered or Roman-numeral section labels;
97
+ - every section centered or symmetrically weighted;
98
+ - the same font for headings, prose, code, and tabular data;
99
+ - full-width primary buttons by default;
100
+ - explanatory text that narrates a bug fix or implementation history;
101
+ - visible release/version marketing in the public application.
102
+
103
+ ## Review checklist
104
+
105
+ Before changing the public UI, check:
106
+
107
+ 1. Does the change improve operating or reading the research tool?
108
+ 2. Is it using an existing type, color, spacing, and button role?
109
+ 3. Can a border/card/help paragraph be removed without losing meaning?
110
+ 4. Does the result remain readable in dark and light appearance?
111
+ 5. Did we preserve exact field/button terminology used by validation docs?
112
+ 6. Did we avoid spending ZeroGPU quota merely to test an unchanged inference path?
README.md CHANGED
@@ -11,297 +11,102 @@ pinned: false
11
  license: mit
12
  ---
13
 
14
- # FeatureLens — Causal Interpretability Workbench
15
 
16
- > **v0.14:** an offline-study transition release: prompt-wide held-out SAE discovery, activation-resample stability, cross-concept association-vs-causality synthesis, a resume-safe study runner, and an artifact-backed public study dashboard.
17
 
18
- **Research question:**
19
 
20
- > Do sparse features that predict a concept also causally influence model behaviour?
21
 
22
- FeatureLens uses **Qwen3-1.7B-Base** with **Qwen-Scope residual-stream sparse autoencoders (SAEs)**. It is not an SAE viewer clone: the project separates representation quality, held-out predictiveness, robustness, and causal intervention evidence.
23
 
 
24
 
25
- ## Evidence ladder
 
 
 
 
 
 
 
 
 
 
 
 
 
26
 
27
- 1. **Reconstruction** does the SAE represent the residual stream reasonably well?
28
- 2. **Prediction** — does a feature distinguish a controlled concept on held-out paraphrase groups?
29
- 3. **Robustness** — does the representation survive rewording locally and prompt-wide?
30
- 4. **Candidate discovery** — which features are more active for a chosen controlled concept than for the other groups?
31
- 5. **Controlled contrast** — does a selected feature prefer one concept group in a balanced live mini-batch?
32
- 6. **Cue sensitivity** — is a feature responding to a lexical/completion cue rather than the underlying concept?
33
- 7. **Candidate causal specificity** — do shortlisted candidate ablations exceed their own norm-matched random perturbation ensembles?
34
- 8. **Single-feature causality** — does ablation/scaling change downstream probability?
35
- 9. **Dose-response** — does effect size vary coherently with feature coefficient?
36
- 10. **Feature-set causality** — do small sparse subspaces matter more than one feature?
37
- 11. **Non-additivity** — does a joint edit differ from the sum of individual effects?
38
- 12. **Specificity** — are targeted effects larger than norm-matched random residual perturbations?
39
- 13. **Contrastive specificity** — does an edit change preference between a desired and comparison continuation?
40
- 14. **Geometry** — are selected SAE decoder directions aligned, orthogonal, or cancelling before downstream model non-linearity?
41
- 15. **Discovery stability** — does a live candidate shortlist survive a split-half check inside the same activation batch?
42
- 16. **Cross-target profile** — does a candidate ablation concentrate on one exact continuation or move several alternatives?
43
- 17. **Controlled evidence patterns** — is random-normalized influence target-weighted, distribution-shift weighted, broad, or weak/mixed?
44
- 15. **Association–causality concordance** — do features ranked highly by concept evidence also rank highly by downstream target effect or distribution shift?
45
 
46
- A high AUROC, high paraphrase overlap, or a large activation is still correlational evidence. Causal evidence comes from downstream change under controlled intervention.
47
 
48
- ## Causal baseline discipline
49
-
50
- v0.3 batched multiple residual edits for efficiency, but compared them with a **separately executed baseline**. On real Qwen3 inference this produced a small numerical discrepancy in the nominal `1×` no-edit dose-response row.
51
-
52
- FeatureLens therefore includes an explicit **batched zero-edit reference** in causal batches. Every effect inside that batch is measured against the zero-edit row from the **same execution context**.
53
-
54
- For the scale dose-response, the `1×` row itself is the no-edit reference:
55
-
56
- ```text
57
- 0× = full ablation
58
- 0.5× = half native coefficient
59
- 1× = batched no-edit reference
60
- 1.5× = 1.5 times native coefficient
61
- 2× = double native coefficient
62
- 3× = triple native coefficient
63
- ```
64
-
65
- The UI also reports single-forward → batched-null execution drift as a diagnostic instead of silently treating it as a causal effect.
66
-
67
- ## Random-control ensemble
68
-
69
- A single random residual direction can be unusually weak or unusually disruptive by chance. FeatureLens uses **8 deterministic norm-matched random directions**.
70
-
71
- For a targeted residual delta `Δh`, each control satisfies:
72
-
73
- ```text
74
- ||Δh_random||₂ = ||Δh_SAE||₂
75
- ```
76
-
77
- The app reports:
78
-
79
- - random signed mean effect;
80
- - random mean absolute effect;
81
- - random-effect standard deviation;
82
- - targeted/random magnitude ratio;
83
- - an exploratory empirical tail probability;
84
- - equivalent next-token Jensen-Shannon statistics.
85
-
86
- The live empirical probability is intentionally coarse with only eight controls. The offline benchmark aggregates across held-out tasks and control ensembles.
87
-
88
- ## Live application
89
-
90
- ### Workbench
91
-
92
- The main workflow supports:
93
-
94
- - prompt entry with explicit **Prompt tokens**;
95
- - layers **4, 14, 26**;
96
- - token-local SAE inspection;
97
- - TopK feature activations;
98
- - reconstruction cosine / NMSE / Top-5 activation mass;
99
- - feature ablation, scaling, and injection;
100
- - baseline vs SAE-edited greedy generation;
101
- - next-token distribution shifts;
102
- - exact **full-continuation teacher-forced scoring**;
103
- - per-target-token log-probability decomposition;
104
- - 8-direction norm-matched random controls;
105
- - batched scale dose-response with its own feature id and target continuation;
106
- - in-place plot/table focus that preserves the original plot aspect ratio and page location.
107
-
108
- ### Feature sets
109
-
110
- FeatureLens can jointly ablate or scale selected active features:
111
 
112
  ```text
113
- h' = h + Σᵢ Δzᵢ dᵢ
 
 
114
  ```
115
 
116
- The tab includes:
117
-
118
- - manual joint feature-set intervention;
119
- - automatic **1 / 3 / 5 strongest-feature ablation sweep**;
120
- - random-ensemble specificity for each set size;
121
- - **individual-vs-joint non-additivity decomposition** for 2–5 selected features.
122
 
123
- For the interaction decomposition, FeatureLens compares:
124
-
125
- ```text
126
- observed joint effect
127
- vs
128
- sum of individual feature effects
129
- ```
130
 
131
- The difference is reported as an **interaction excess**. A non-zero excess means the downstream response is non-additive under these edits; it does not by itself prove a direct feature-feature circuit.
132
 
133
- ### Feature evidence
134
 
135
- The **Feature evidence** tab supports a discovery-to-causality workflow:
136
 
137
- 1. **Concept-guided candidate discovery** — choose one of the seven controlled concept groups and rank SAE features with Balanced selectivity, Causal-ready-at-current-token, or raw mean-difference evidence. The result is a candidate list, **not a semantic label**.
138
- 2. **Batched causal candidate triage** — ablate up to eight discovered candidates in one scoring batch and rank their target effects before spending the full random-control ensemble on a selected feature.
139
- 3. **Feature-token activation trace** — show exactly where a selected feature enters the SAE TopK support across every token of the current Workbench prompt.
140
- 4. **Completion-cue sensitivity** — append controlled suffixes/cues to one prompt stem and measure the selected feature at the final token.
141
- 5. **Cue × context specificity** — cross cues with unrelated stems; a strong tested **cue-dominant** pattern is reported explicitly when one cue activates across all tested contexts and off-cue conditions stay inactive.
142
- 6. **Prompt-wide controlled concept contrast** — evaluate one selected feature on a small balanced batch from:
143
 
144
- - code;
145
- - mathematics;
146
- - positive sentiment;
147
- - negative sentiment;
148
- - German language;
149
- - factual entities;
150
- - uncertainty.
 
151
 
152
- Each controlled prompt is summarized by the feature's **maximum activation across non-padding tokens**. Candidate discovery defaults to **Balanced selectivity** — `positive selectivity × target activation rate × log1p(target mean activation)` — so globally large but weakly selective coefficients do not dominate purely by scale. A **Raw mean difference** mode remains available for comparison. The discovery table also reports current-Workbench prompt/token activation so candidates can be separated from candidates that are immediately causal-testable at the selected location. The controlled contrast reports mean/median prompt-wide maxima, prompt activation rate, mean activation when active, and maximum activation by concept.
153
 
154
- Both tools are exploratory live evidence. They never overwrite `Offline concept hint`; the held-out offline AUROC/F1 benchmark remains the stronger feature/concept claim.
155
-
156
- ### Contrastive continuation preference
157
-
158
- A probability increase for one target can reflect a broad distribution shift rather than selective behavioral control. FeatureLens includes a stricter live test with two exact continuations, A and B. FeatureLens scores both teacher-forced under the same SAE intervention and random-control ensemble, then reports the causal change in:
159
-
160
- ```text
161
- log P(A) - log P(B)
162
  ```
163
 
164
- The app also reports a token-normalized preference shift, targeted/random magnitude ratio, and exploratory empirical tail probability. This lets the workbench distinguish **general distributional influence** from a more selective change in relative continuation preference.
165
-
166
- ### Feature-set decoder geometry
167
-
168
- The Feature Sets tab can inspect 2–8 selected decoder directions without an additional generation experiment. It reports pairwise decoder cosine similarities and compares the L2 norm of the activation-weighted joint ablation with the root-sum-square norm expected for independent directions. A ratio below 1 indicates net geometric cancellation; above 1 indicates net alignment. This is a geometric diagnostic, not by itself evidence of downstream causal interaction.
169
-
170
- ### Paraphrase robustness
171
-
172
- FeatureLens reports two complementary views:
173
-
174
- 1. **Selected-token robustness** — TopK Jaccard and sparse cosine at the two manually chosen token positions.
175
- 2. **Prompt-wide robustness** — for each SAE feature, take its maximum activation across all prompt tokens, then compare the two sparse prompt profiles.
176
-
177
- The prompt-wide view reduces sensitivity to accidentally comparing semantically different final tokens.
178
-
179
- ### Layer trajectory
180
-
181
- The trajectory view compares SAE reconstruction and sparsity/concentration diagnostics at the same prompt token across layers 4, 14 and 26. It does **not** equate feature IDs across layer-specific SAE dictionaries.
182
 
183
- ## Interface and navigation
184
-
185
- The interface is designed so a reviewer does not need prior mechanistic-interpretability vocabulary to find the controls:
186
-
187
- - a **Start here** tab explains the evidence workflow and glossary in plain language;
188
- - a persistent **Current Workbench context** banner states the active prompt, layer and token after inspection;
189
- - experiments that reuse that state explicitly say **Uses current Workbench context**;
190
- - single-feature causal, dose-response, contrastive-preference, feature-set and evidence panels expose their own editable feature selectors instead of silently borrowing a hidden value;
191
- - concept-guided discovery gives users a route to candidate features without knowing feature IDs in advance;
192
- - headings now follow a conventional descending hierarchy and table headers use the same readable base scale as the surrounding interface;
193
- - tables and plots use a bounded **focus mode** instead of stretching to the entire viewport; plot exports are renamed from the generic browser filename to descriptive `featurelens_*.png` names;
194
- - all action/copy buttons retain one muted-teal visual language and copy actions provide visible confirmation.
195
-
196
- The centered 1600 px research canvas, bounded result-table heights, dynamic resize observer and explicit bottom padding from v0.5 remain in place for Hugging Face's embedded Space layout.
197
-
198
- ## Copy tables with headers
199
-
200
- Every result table that is intended for analysis has an explicit **Copy table with headers** button. The app serializes the displayed data as TSV before copying, so column names are preserved.
201
-
202
- The Dataframe focus control is retained, while the custom copy action avoids relying on browser text selection and gives immediate visual confirmation when clipboard writing succeeds.
203
-
204
- ## Reconstruction-preserving edits
205
-
206
- Let `h` be the original residual vector, `z_i` the SAE coefficient, and `d_i` the decoder direction.
207
-
208
- ```text
209
- ablate: h' = h - z_i d_i
210
- scale α: h' = h + (α - 1) z_i d_i
211
- inject δ: h' = h + δ d_i
212
  ```
213
 
214
- FeatureLens patches only the decoder-direction **delta** into the original residual. It never replaces the residual with the full SAE reconstruction, avoiding reconstruction error as a causal confound.
215
-
216
- ## Full-continuation scoring
217
-
218
- A target such as `2x` can tokenize into more than one token. FeatureLens concatenates the exact target token IDs to the prompt and scores every target token teacher-forced.
219
-
220
- It reports:
221
-
222
- - target sequence log probability;
223
- - mean target log probability per token;
224
- - per-token log probabilities;
225
- - targeted edit deltas;
226
- - random-ensemble comparison.
227
-
228
- Greedy generations can remain identical while these probability-level metrics move. A deterministic text change only occurs after an edit crosses an argmax boundary.
229
-
230
- ## Offline experiment
231
-
232
- The repository ships:
233
-
234
- - **224 discovery prompts**;
235
- - **112 paraphrase pairs**;
236
- - 7 controlled concepts;
237
- - **28 separate causal completion tasks**;
238
- - residual/SAE activation collection at layers 4, 14 and 26.
239
-
240
- The offline pipeline computes:
241
-
242
- - SAE reconstruction cosine / NMSE;
243
- - held-out feature/concept AUROC and F1;
244
- - paraphrase TopK Jaccard and sparse activation cosine;
245
- - dense residual multinomial linear-probe baselines;
246
- - selected-feature ablation and 2× amplification;
247
- - exact full-target mean log probability per token;
248
- - next-token JS divergence and top-1 changes;
249
- - **random-control ensembles** rather than one arbitrary random direction;
250
- - top-1 / top-3 / top-5 joint feature-set ablations;
251
- - bootstrap 95% confidence intervals;
252
- - paired sign-flip tests.
253
-
254
- Feature selection is performed using the **training split only**. Paraphrase pairs never cross the train/test boundary.
255
-
256
- ## Run the offline study
257
-
258
- A CUDA machine is strongly recommended for activation collection and causal intervention stages. v0.14 changes the offline SAE concept representation to **prompt-wide max-pooled activation across all non-padding prompt tokens**. Final-token sparse activations are saved separately for local diagnostics; the dense residual linear probe remains a final-token baseline.
259
 
260
  ```bash
261
- python -m venv .venv
262
- source .venv/bin/activate
263
- pip install -r requirements.txt
264
- python experiments/run_all.py
265
  ```
266
 
267
- For interruptible/preemptible GPU sessions, use:
268
 
269
  ```bash
270
- python experiments/run_all.py --resume
271
  ```
272
 
273
- Completed stages are skipped when their expected artifacts already exist. After the expensive model/SAE stages have completed, all CPU-only evaluation/report logic can be rerun without another inference pass:
274
 
275
- ```bash
276
- python experiments/run_analysis_only.py
277
- ```
278
 
279
- The full pipeline is:
280
 
281
- ```text
282
- build_dataset
283
- → collect_activations
284
- ├─ prompt-wide SAE feature maxima
285
- └─ final-token sparse features + final-token residuals
286
- → evaluate_features
287
- ├─ held-out AUROC/F1
288
- ├─ dense residual linear probe
289
- └─ prompt-wide paraphrase stability
290
- → run_causal
291
- → run_feature_sets
292
- → analyze_stability
293
- └─ 128 balanced activation resamples
294
- → analyze_study
295
- └─ association/stability ↔ random-normalized causal evidence
296
- → make_report
297
- → validate_artifacts
298
- ```
299
 
300
- Outputs are generated under `artifacts/`:
301
 
302
  ```text
303
  artifacts/
304
- ├── activations/ # large; keep gitignored
305
  ├── feature_catalog.csv
306
  ├── layer_metrics.csv
307
  ├── stability.csv
@@ -313,175 +118,66 @@ artifacts/
313
  ├── summary.json
314
  ├── report.md
315
  └── figures/
316
- ├── feature_auroc.png
317
- ├── layer_diagnostics.png
318
- ├── causal_effects.png
319
- ├── feature_set_effects.png
320
- ├── association_vs_causality.png
321
- └── candidate_stability.png
322
- ```
323
-
324
- Validate the small public artifacts before committing them:
325
-
326
- ```bash
327
- python -m scripts.validate_artifacts
328
- ```
329
-
330
- `summary.json`, `study_summary.json`, and `report.md` are generated from measured artifacts. No scientific benchmark numbers are fabricated in the repository. The live **Offline study** tab automatically turns into a results dashboard once these small CSV/JSON/figure artifacts are committed.
331
-
332
- See [`docs/OFFLINE_STUDY.md`](docs/OFFLINE_STUDY.md) for the staged workflow and interpretation guardrails.
333
-
334
- ## Hugging Face deployment
335
-
336
- FeatureLens is a **Gradio SDK Space**. The app keeps SSR disabled and loads only SAE layers 4, 14 and 26 for the live workbench.
337
-
338
- Useful environment variables:
339
-
340
- ```text
341
- FEATURELENS_MODEL_ID=Qwen/Qwen3-1.7B-Base
342
- FEATURELENS_SAE_REPO=Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50
343
- FEATURELENS_LAYERS=4,14,26
344
- FEATURELENS_EAGER_LOAD=1
345
- FEATURELENS_SAE_DTYPE=float16
346
- FEATURELENS_MAX_NEW_TOKENS=32
347
- FEATURELENS_LIVE_RANDOM_CONTROLS=8
348
- FEATURELENS_CONTRAST_PROMPTS_PER_CONCEPT=4
349
  ```
350
 
351
- ## UI design
352
-
353
- The interface deliberately stays closer to a conventional research instrument than a rounded dashboard:
354
-
355
- - Times / Liberation Serif-style formal typography;
356
- - restrained teal, umber, stone, plum, and muted red chart accents;
357
- - flatter controls and smaller corner radii;
358
- - aligned intervention forms;
359
- - explicit section rules instead of pill badges;
360
- - extra bottom padding and a visible end-of-workbench footer to avoid an abrupt embedded-page cutoff.
361
-
362
- The app also installs a lightweight browser-side reflow observer for dynamic output-height changes. Hugging Face still owns the outer embedding frame, so the direct Space URL remains a useful diagnostic if embedded-page scrolling ever behaves differently.
363
 
364
  ## Repository layout
365
 
366
  ```text
367
  FeatureLens/
368
- ├── app.py
369
- ├── featurelens/
370
- ├── config.py
371
- ├── sae.py
372
- ├── interventions.py
373
- ├── runtime.py
374
- ├── metrics.py
375
- ├── stats.py
376
- ├── selection.py
377
- ├── catalog.py
378
- └── study.py
379
- ├── experiments/
380
- │ ├── build_dataset.py
381
- │ ├── collect_activations.py
382
- │ ├── evaluate_features.py
383
- │ ├── run_causal.py
384
- │ ├── run_feature_sets.py
385
- │ ├── analyze_stability.py
386
- │ ├── analyze_study.py
387
- │ ├── make_report.py
388
- │ ├── run_analysis_only.py
389
- │ └── run_all.py
390
- ├── data/
391
- ├── tests/
392
- ├── scripts/
393
- │ ├── release_check.py
394
- │ ├── validate_artifacts.py
395
- │ └── ui_smoke.py
396
- ├── docs/
397
- └── research_config.json
398
  ```
399
 
400
- ## v0.10 association-to-causality synthesis
401
-
402
- - **Discovery–causality alignment** joins the concept-discovery table with the batched ablation screen for the same shortlist.
403
- - **Three ranks are kept separate:** discovery evidence, target-continuation effect, and whole next-token distribution shift.
404
- - **Descriptive Spearman concordance** reports candidate-score correlation with absolute target effect and with next-token JS; small live screens are never presented as significance tests.
405
- - **Rank-shift diagnostics** make cases such as a discovery-rank #1 feature becoming target-effect rank #5 immediately visible.
406
- - **Association-vs-target-effect scatter** is generated from the same two existing GPU calls; the synthesis itself costs no additional GPU time.
407
- - Existing v0.9 causal-ready discovery, candidate triage, in-place focus, and GPU-budget-aware validation remain unchanged.
408
-
409
- ## v0.11 controlled candidate specificity
410
-
411
- - **Controlled candidate specificity** compares up to three shortlisted SAE ablations against each candidate's own eight-direction norm-matched random ensemble in one shared batched execution context.
412
- - **Strategic shortlist selection** preserves the discovery leader, target-effect leader, and distribution-shift leader when they differ, then fills remaining slots by triage target rank. This avoids confirming only whichever metric already won.
413
- - **Two controlled causal notions remain separate:** target-specificity ratio normalizes the specified continuation effect; JS-specificity ratio normalizes whole next-token distribution shift.
414
- - **Association vs controlled causality** joins discovery score/rank to random-normalized target and JS specificity, with descriptive Spearman concordance and rank shifts.
415
- - The controlled live screen remains deliberately small: three candidates × eight controls each plus one zero-edit reference. Empirical tails are therefore coarse diagnostics rather than conventional significance tests.
416
- - The validated in-place focus behavior and unchanged paraphrase/trajectory/feature-set paths are not touched. HF acceptance for v0.11 requires only the new controlled-candidate GPU path.
417
-
418
- ## v0.12 evidence synthesis and cross-target profiling
419
-
420
- v0.12 bundles several related improvements instead of adding one isolated widget:
421
-
422
- - **Split-half discovery stability** reuses the concept-discovery activation batch and compares candidate shortlists from two prompt halves. This adds no GPU inference and is explicitly a small-sample sensitivity diagnostic rather than a reliability estimate.
423
- - **Controlled evidence patterns** always summarize the random-normalized candidate table, separating broad controlled influence, target-weighted effects, distribution-shift-dominant effects, and weak/mixed specificity. These are effect-ratio descriptions, not significance labels.
424
- - **Association vs controlled causality** no longer fails silently when discovery state is absent after a Space rebuild. The UI explains that controlled evidence remains valid while discovery-rank alignment requires the discovery table from the current browser session.
425
- - **Cross-target causal profile** screens up to three candidate ablations across two to five exact continuations. It reports target-wise mean/sequence log-probability deltas, next-token JS, the strongest target per feature, and a target-profile ratio. This stage intentionally omits random controls; controlled candidate specificity remains the matched-random causal test.
426
- - HF validation remains GPU-budget-aware: rerun only the touched discovery path and the new cross-target path. Unchanged paraphrase, trajectory, set-size, dose-response, cue, and focus paths stay covered by automated tests.
427
-
428
- ## v0.14 offline-study transition
429
-
430
- v0.14 deliberately stops expanding the live intervention surface and strengthens the empirical study behind it.
431
-
432
- - **Prompt-wide offline SAE evidence:** concept-feature discovery and paraphrase stability use each feature's maximum activation across all non-padding prompt tokens. This avoids treating an arbitrary final token as the semantic representation of an entire prompt. Final-token sparse activations remain available separately for local diagnostics.
433
- - **Activation-resample candidate selection stability:** `analyze_stability.py` performs 128 deterministic balanced resamples of the saved prompt-wide activations and records shortlist support plus median/mean resample rank. No model inference is repeated.
434
- - **Association vs random-normalized causality across concepts:** `analyze_study.py` joins held-out AUROC/F1, paraphrase stability, candidate resample support, causal active rate, target specificity, and JS specificity for the selected feature of every controlled concept. Cross-concept Spearman correlations are descriptive because there are only seven concepts.
435
- - **Artifact-backed Offline study tab:** the public app remains honest when no benchmark has been run, then automatically displays committed study tables and figures when real artifacts exist.
436
- - **Resume-safe full runner:** `python experiments/run_all.py --resume` skips completed stages; `python experiments/run_analysis_only.py` reruns only CPU evaluation/stability/report logic once expensive inference artifacts exist.
437
- - **Artifact schema validation:** `python -m scripts.validate_artifacts` checks that public study outputs are non-empty, prompt-wide v0.14 activations were used, and required causal/stability columns exist.
438
-
439
- ## Validation
440
 
441
  ```bash
442
- python -m ruff check app.py featurelens experiments tests scripts
443
- python -m pytest -q
444
- python -m compileall -q app.py featurelens experiments scripts
445
- python scripts/ui_smoke.py
446
- python scripts/release_check.py
447
  ```
448
 
449
- See [`docs/VALIDATION.md`](docs/VALIDATION.md). The v0.12 guide uses the **exact labels shown in the UI** and asks for only two touched/new HF paths: causal-ready discovery (to validate split-half stability) and the cross-target causal profile. Unchanged paraphrase, trajectory, feature-set, dose-response, cue, controlled-specificity, and focus paths are not rerun.
450
-
451
- ## Limitations
452
-
453
- - SAE features are sparse directions, not guaranteed monosemantic concepts.
454
- - Live candidate discovery and concept contrast are small controlled diagnostics, not a universal ontology or semantic labeling system.
455
- - Completion-cue sensitivity diagnoses lexical/structural dependence but does not establish a feature's full semantics.
456
- - Prompt-wide max pooling measures feature presence anywhere in the prompt and discards token order.
457
- - Joint residual edits can be non-additive downstream without implying direct feature-feature interaction.
458
- - Eight live random controls give only a coarse empirical tail probability.
459
- - Teacher-forced target scoring measures probability assigned to a specified continuation, not sampled free-running sequence probability.
460
- - All causal effects remain prompt-, token-, layer-, feature-, and intervention-scale dependent.
461
-
462
- ## Resume-ready description
463
 
464
- > **FeatureLens — Causal Interpretability Workbench** | PyTorch, Qwen3, Sparse Autoencoders, Mechanistic Interpretability, Gradio
465
- > Built an SAE-based interpretability system for Qwen3-1.7B with held-out concept discovery, concept-guided candidate discovery, token/prompt-wide, completion-cue, and cue × context feature evidence, reconstruction-preserving single and multi-feature interventions, full-continuation and contrastive preference scoring, dose-response analysis, decoder-geometry/non-additivity diagnostics, discovery-to-causality rank analysis, multi-candidate random-controlled specificity screening, split-half/resample discovery stability, cross-target causal profiling, controlled evidence-pattern synthesis, prompt-wide held-out SAE evaluation, association-vs-random-normalized-causality study synthesis, and norm-matched random-control ensembles.
466
 
467
- ## Acknowledgements
468
 
469
- FeatureLens builds on the open Qwen3 model and Qwen-Scope residual-stream SAE checkpoints from the Qwen team.
 
 
 
 
 
 
470
 
471
- ## v0.13 stability and target-selectivity synthesis
472
 
473
- v0.13 deliberately adds **no new GPU callback**. It strengthens the two live paths already used in v0.12.
474
-
475
- ### Resample shortlist support
476
-
477
- Concept-guided discovery still reports split-half Jaccard, but now also reranks the same already-computed controlled activations under **32 deterministic balanced bootstrap resamples**. Each displayed feature receives a **Resample shortlist support** fraction and **Median resample rank**. The summary reports the mean displayed support and how many candidates survive at least 75% of resampled shortlists. These are small-sample sensitivity diagnostics, not confidence intervals or held-out reliability claims.
478
 
479
- ### Cross-target effect concentration
 
 
 
 
 
480
 
481
- The cross-target profile now separates a feature whose causal effect is concentrated on one exact continuation from one that broadly moves several alternatives. For each feature it reports **Normalized effect entropy**, **Effect concentration = 1 − entropy**, and **Signed bias = ΣΔ / Σ|Δ|**. A descriptive profile pattern labels cases such as target-concentrated/mixed-sign or broad same-sign suppression/enhancement; this label is a heuristic summary, not a semantic feature label.
482
 
483
- ### Pairwise target preference
484
 
485
- Every cross-target run also derives **pairwise target preference shifts** without another forward pass. For targets A and B, FeatureLens reports `Δ(A−B) = Δ mean log p/token(A) − Δ mean log p/token(B)`. Positive values mean the SAE ablation shifts token-normalized preference toward A relative to B. This turns the same cross-target scores into a more direct comparison of alternatives while keeping the random-controlled specificity experiment separate.
486
 
487
- See [`docs/VALIDATION.md`](docs/VALIDATION.md). v0.13 requires only the two touched GPU paths: causal-ready discovery and cross-target profiling.
 
11
  license: mit
12
  ---
13
 
14
+ # FeatureLens
15
 
16
+ FeatureLens is a causal interpretability workbench for `Qwen/Qwen3-1.7B-Base` and the **Qwen-Scope residual-stream sparse autoencoders**. It is built around one question:
17
 
18
+ > **Do sparse features that predict a concept also causally influence model behaviour?**
19
 
20
+ The project keeps association and intervention evidence separate. A large SAE activation or strong held-out classifier is useful evidence about representation; a causal claim requires changing the residual stream and measuring downstream behaviour against controlled perturbations.
21
 
22
+ ## What the live app does
23
 
24
+ FeatureLens loads SAEs for residual layers **4, 14, and 26** and supports:
25
 
26
+ - token-level residual capture and TopK SAE feature inspection;
27
+ - reconstruction cosine, NMSE, activation mass, and layer trajectories;
28
+ - single-feature ablation, scaling, and decoder-direction injection;
29
+ - exact full-continuation teacher-forced scoring;
30
+ - next-token distribution shifts and deterministic generation comparison;
31
+ - eight norm-matched random controls for live specificity checks;
32
+ - scale dose-response curves;
33
+ - contrastive continuation preference;
34
+ - joint feature-set interventions, 1/3/5 set-size sweeps, non-additivity, and decoder geometry;
35
+ - concept-guided candidate discovery with current-token causal readiness;
36
+ - batched candidate triage and random-controlled candidate comparison;
37
+ - token traces, completion-cue tests, cue × context tests, and controlled concept contrasts;
38
+ - local and prompt-wide paraphrase robustness;
39
+ - cross-target causal profiles and pairwise preference shifts.
40
 
41
+ The interface is deliberately an analytical tool rather than an SAE label viewer. Feature ids remain unlabeled until there is empirical evidence for a concept association.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42
 
43
+ ## Intervention semantics
44
 
45
+ For residual vector `h`, SAE coefficient `z_i`, decoder direction `d_i`, and multiplier `α`:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
 
47
  ```text
48
+ ablate: h' = h - z_i d_i
49
+ scale: h' = h + (α - 1) z_i d_i
50
+ inject: h' = h + δ d_i
51
  ```
52
 
53
+ FeatureLens applies the decoded **delta** to the original residual. It does not replace the residual with the full SAE reconstruction, so reconstruction error is not silently mixed into the intervention.
 
 
 
 
 
54
 
55
+ Batched causal experiments include a zero-edit condition in the same execution context. Random controls match the L2 norm of the SAE perturbation.
 
 
 
 
 
 
56
 
57
+ ## Offline study
58
 
59
+ The live app is exploratory. The offline study is the dataset-scale experiment.
60
 
61
+ It uses **224 discovery prompts** arranged as 112 paraphrase pairs across seven controlled concepts, plus **28 separate causal tasks**. Concept evidence uses prompt-wide max-pooled SAE activations across non-padding tokens; final-token sparse activations are saved separately for local analyses.
62
 
63
+ The study produces:
 
 
 
 
 
64
 
65
+ - train-only feature selection with held-out AUROC/F1;
66
+ - a dense final-token residual linear-probe baseline;
67
+ - paraphrase stability;
68
+ - 128-resample candidate-selection sensitivity;
69
+ - random-controlled single-feature causal results;
70
+ - top-1/3/5 feature-set causal results;
71
+ - cross-concept association-versus-causality synthesis;
72
+ - uncertainty-aware report figures and a measured Markdown report.
73
 
74
+ Run the full pipeline with:
75
 
76
+ ```bash
77
+ python -m experiments.run_all --resume
 
 
 
 
 
 
78
  ```
79
 
80
+ On a memory-constrained GPU, activation collection can be tuned without changing the experiment definition:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
81
 
82
+ ```bash
83
+ python -m experiments.run_all --resume --activation-batch-size 8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
  ```
85
 
86
+ After the expensive model stages exist, CPU-only analysis can be regenerated with:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
87
 
88
  ```bash
89
+ python -m experiments.run_analysis_only
 
 
 
90
  ```
91
 
92
+ Artifact integrity is checked with:
93
 
94
  ```bash
95
+ python -m scripts.validate_artifacts
96
  ```
97
 
98
+ ### Google Colab
99
 
100
+ A ready-to-run notebook is included at [`notebooks/FeatureLens_Offline_Study_Colab.ipynb`](notebooks/FeatureLens_Offline_Study_Colab.ipynb). It mounts Google Drive for persistent artifacts, keeps the Hugging Face cache on the Colab VM, chooses a conservative activation batch from available VRAM, and runs the resumable pipeline.
 
 
101
 
102
+ See [`docs/COLAB.md`](docs/COLAB.md) for the exact workflow.
103
 
104
+ ## Public artifacts
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
 
106
+ Large activation caches are intentionally excluded from Git. The small measured outputs that can be committed after a real run include:
107
 
108
  ```text
109
  artifacts/
 
110
  ├── feature_catalog.csv
111
  ├── layer_metrics.csv
112
  ├── stability.csv
 
118
  ├── summary.json
119
  ├── report.md
120
  └── figures/
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
  ```
122
 
123
+ The **Study** tab reads these artifacts directly. Before the offline run is materialized it intentionally shows no placeholder metrics.
 
 
 
 
 
 
 
 
 
 
 
124
 
125
  ## Repository layout
126
 
127
  ```text
128
  FeatureLens/
129
+ ├── app.py # Gradio / ZeroGPU workbench
130
+ ├── featurelens/ # SAE, runtime, metrics, interventions, study loader
131
+ ├── experiments/ # offline collection, evaluation, causal study, reports
132
+ ├── data/ # controlled prompt and causal-task definitions
133
+ ├── artifacts/ # small public study outputs; activations are ignored
134
+ ├── notebooks/ # Colab runner
135
+ ├── scripts/ # release, UI smoke, artifact validation
136
+ ├── tests/ # software and methodology regression tests
137
+ ├── docs/ # methodology, validation, deployment, Colab notes
138
+ ├── DESIGN.md # UI design contract
139
+ └── research_config.json # experiment configuration
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
140
  ```
141
 
142
+ ## Local validation
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
143
 
144
  ```bash
145
+ python3 -m pytest -q
146
+ python3 -m compileall -q app.py featurelens experiments scripts
147
+ python3 -m ruff check app.py featurelens experiments tests scripts
148
+ python3 scripts/ui_smoke.py
149
+ python3 scripts/release_check.py
150
  ```
151
 
152
+ The UI smoke test performs a real local Gradio `launch()` rather than only constructing the component tree.
 
 
 
 
 
 
 
 
 
 
 
 
 
153
 
154
+ ## Methodology notes
 
155
 
156
+ Several quantities answer different questions and should not be collapsed into one score:
157
 
158
+ - **held-out AUROC/F1** concept association;
159
+ - **paraphrase / resample stability** — sensitivity to wording or sample choice;
160
+ - **target Δ log p** — effect on one specified continuation;
161
+ - **Jensen-Shannon divergence** — local distributional change;
162
+ - **random-normalized specificity** — whether the targeted SAE edit exceeds an equal-norm residual perturbation baseline;
163
+ - **feature-set non-additivity** — downstream interaction under joint edits;
164
+ - **decoder geometry** — alignment/cancellation before downstream model non-linearity.
165
 
166
+ The full methodology is documented in [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md) and the offline study protocol in [`docs/OFFLINE_STUDY.md`](docs/OFFLINE_STUDY.md).
167
 
168
+ ## Limitations
 
 
 
 
169
 
170
+ - Live ZeroGPU controls are intentionally small; eight-control empirical tails are coarse diagnostics.
171
+ - SAE feature ids are layer-specific and should not be compared across layers by id.
172
+ - Prompt-wide max pooling discards token order.
173
+ - Dense linear probes and prompt-wide SAE features use different pooling schemes and are reported as separate baselines.
174
+ - Cross-concept study correlations have only seven concepts and are descriptive.
175
+ - A candidate feature can be predictive without being causally specific, and a causally disruptive feature need not selectively control the target one might infer from its association.
176
 
177
+ ## Design
178
 
179
+ The public UI follows the project-specific design contract in [`DESIGN.md`](DESIGN.md): restrained typography and color, flat information hierarchy, minimal decorative chrome, compact actions, explicit table headings, and no marketing-style cards/badges/gradients.
180
 
181
+ ## License
182
 
183
+ MIT. See [`LICENSE`](LICENSE).
app.py CHANGED
@@ -9,25 +9,38 @@ from featurelens.runtime import RUNTIME
9
  from featurelens.study import OfflineStudy
10
 
11
  # Restrained, print-inspired palette. The app deliberately avoids saturated dashboard colors.
12
- INK_TEAL = "#708B86"
13
- INK_UMBER = "#94785D"
14
- INK_RED = "#986D68"
15
- INK_PLUM = "#82768F"
16
- INK_STONE = "#8E8A83"
17
- INK_BLUEGREY = "#71808A"
18
 
19
  STUDY = OfflineStudy()
20
 
21
  CSS = r"""
 
 
 
 
 
 
 
 
22
  .gradio-container {
23
- width: min(96vw, 1600px) !important;
24
- max-width: 1600px !important;
25
- margin-left: auto !important;
26
- margin-right: auto !important;
27
- padding: 0 22px 128px !important;
28
- font-family: "Times New Roman", Times, "Liberation Serif", Georgia, serif !important;
29
- font-size: 16px !important;
30
- line-height: 1.52;
 
 
 
 
 
31
  }
32
  .gradio-container input,
33
  .gradio-container textarea,
@@ -36,152 +49,251 @@ CSS = r"""
36
  .gradio-container label,
37
  .gradio-container table,
38
  .gradio-container .prose {
39
- font-family: "Times New Roman", Times, "Liberation Serif", Georgia, serif !important;
40
  }
41
- .gradio-container input,
42
- .gradio-container textarea,
43
- .gradio-container select { font-size: 15.5px !important; }
44
- .gradio-container table { font-size: 15.5px !important; }
45
- .gradio-container table thead th, .gradio-container [role="columnheader"] { font-size: 16px !important; font-weight: 700 !important; }
46
- .gradio-container table tbody td, .gradio-container [role="gridcell"] { font-size: 15.5px !important; }
47
- .gradio-container label, .gradio-container .label-wrap { font-size: 15px !important; }
48
- .gradio-container p, .gradio-container li { font-size: 16px; }
49
- .gradio-container h2 { font-size: 1.72rem; line-height: 1.25; }
50
- .gradio-container h3 { font-size: 1.46rem; line-height: 1.28; }
51
- .gradio-container h4 { font-size: 1.24rem; line-height: 1.30; }
 
 
 
 
 
52
  .hero {
53
- padding: 14px 2px 10px;
54
- border-bottom: 1px solid var(--border-color-primary);
55
- margin-bottom: 14px;
56
  }
57
  .hero h1 {
58
  margin: 0;
59
- font-size: 2.18rem;
 
60
  font-weight: 600;
61
- letter-spacing: 0;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
62
  }
63
- .hero .subtitle { margin-top: 4px; font-size: 1.08rem; opacity: .82; }
64
- .hero .question { margin-top: 6px; font-size: 1rem; opacity: .72; }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  .section-rule {
66
- margin: 20px 0 11px;
67
- padding-top: 9px;
68
- border-top: 1px solid var(--border-color-primary);
69
- font-variant: small-caps;
70
- letter-spacing: .045em;
71
- font-size: 1.12rem;
72
- font-weight: 700;
73
- opacity: .88;
74
  }
75
- .token-wrap { display: flex; flex-wrap: wrap; gap: 6px; padding: 8px 2px 14px; line-height: 1.9; }
76
- .token {
77
- background: var(--background-fill-secondary);
78
- border: 1px solid var(--border-color-primary);
79
- border-radius: 2px;
80
- padding: 3px 7px;
81
- font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace !important;
82
- font-size: 12.5px;
 
83
  }
84
- .token.selected { border: 2px solid #708B86; font-weight: 700; }
85
- .token sup { opacity: .58; margin-right: 4px; }
86
- .small-note { opacity: .74; font-size: 14px; }
 
 
87
  .instrument-note {
88
- border: 1px solid var(--border-color-primary);
89
- border-radius: 2px;
90
- padding: 10px 12px;
91
- background: var(--background-fill-secondary);
92
- margin: 6px 0 11px;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
  }
94
- .copy-row { margin-top: -4px; }
95
- .copy-row button { min-height: 34px !important; }
96
- .gradio-container button { border-radius: 3px !important; font-weight: 600 !important; font-size: 15px !important; }
97
- .action-btn button, .copy-btn button {
98
- background: #5f817b !important;
 
99
  color: #fff !important;
100
- border: 1px solid #5f817b !important;
101
  }
102
- .action-btn button:hover, .copy-btn button:hover { background: #536f6a !important; border-color: #536f6a !important; }
103
- .gradio-container .form, .gradio-container .block { border-radius: 3px !important; }
104
- .gradio-container textarea, .gradio-container input { border-radius: 2px !important; }
105
- .form-note { margin-top: -3px; margin-bottom: 9px; font-size: .92rem; opacity: .72; }
106
- .wide-table { width: 100% !important; }
107
- .bottom-spacer { height: 110px; width: 100%; }
108
- .tabs, .tabitem { padding-bottom: 26px !important; }
109
- .context-card {
110
- border: 1px solid var(--border-color-primary);
111
- border-left: 4px solid #708B86;
112
- background: var(--background-fill-secondary);
113
- padding: 10px 14px;
114
- margin: 8px 0 16px;
115
- border-radius: 2px;
116
  }
117
- .context-card p { margin: 0 !important; }
118
- .start-card {
119
- border: 1px solid var(--border-color-primary);
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
120
  background: var(--background-fill-secondary);
121
- padding: 14px 16px;
122
- min-height: 150px;
123
  border-radius: 2px;
 
 
 
124
  }
125
- .start-card h3 { margin-top: 0; }
126
- .graph-note { font-size: .95rem; opacity: .72; margin-top: 2px; }
 
 
127
  .table-heading {
128
- margin: 0 0 -34px !important;
129
  padding: 4px 58px 0 0 !important;
130
- min-height: 34px;
131
  position: relative;
132
  z-index: 3;
133
  pointer-events: none;
134
- font-size: 1.20rem !important;
135
- font-weight: 700 !important;
136
  line-height: 1.22 !important;
137
  }
138
- .result-table .label-wrap,
139
- .result-table .label-wrap span,
140
- .result-table label,
141
- .result-table label span,
142
- .result-table [data-testid="block-label"],
143
- .result-table [data-testid="block-label"] span,
144
- .result-table .block-label,
145
- .result-table .block-title {
146
- font-size: 19px !important;
147
- font-weight: 700 !important;
148
- line-height: 1.30 !important;
149
  }
150
  .result-table table thead th,
151
  .result-table table thead th *,
152
  .result-table [role="columnheader"],
153
  .result-table [role="columnheader"] * {
154
- font-size: 16.75px !important;
155
- font-weight: 700 !important;
156
- line-height: 1.25 !important;
 
 
 
 
 
 
157
  }
158
- /* Gradio native fullscreen is intercepted and converted to an in-place focus mode.
159
- Nothing is moved to the top of the HF iframe: the original component expands where the user clicked it. */
 
 
 
 
 
 
 
160
  .fl-plot.featurelens-inline-focus,
161
  .result-table.featurelens-inline-focus {
162
  position: relative !important;
163
  z-index: 5000 !important;
164
  background: var(--background-fill-primary) !important;
165
- border: 1px solid var(--border-color-primary) !important;
166
- box-shadow: 0 12px 36px rgba(0, 0, 0, .42) !important;
167
- border-radius: 4px !important;
168
- }
169
- .fl-plot.featurelens-inline-focus {
170
- transform-origin: top left !important;
171
- }
172
- .result-table.featurelens-inline-focus {
173
- overflow: visible !important;
174
  }
175
- .candidate-help { opacity: .78; font-size: .97rem; margin-top: -2px; }
 
 
 
 
 
 
 
 
 
 
176
  @media (max-width: 900px) {
177
  .gradio-container { width: 100% !important; padding-left: 12px !important; padding-right: 12px !important; }
 
 
178
  }
179
  """
180
-
181
  THEME = gr.themes.Base(
182
  primary_hue="teal",
183
- secondary_hue="stone",
184
- neutral_hue="stone",
185
  radius_size="sm",
186
  )
187
 
@@ -193,10 +305,10 @@ COPY_JS = r"""
193
  const oldLabel = button ? button.innerText : null;
194
  const signal = () => {
195
  if (!button) return;
196
- button.innerText = "Copied with headers";
197
  button.disabled = true;
198
  window.setTimeout(() => {
199
- button.innerText = oldLabel || "Copy table with headers";
200
  button.disabled = false;
201
  }, 1200);
202
  };
@@ -362,7 +474,7 @@ def _tsv(frame: pd.DataFrame) -> str:
362
  return frame.to_csv(sep="\t", index=False, lineterminator="\n")
363
 
364
 
365
- def _copy_button(label: str = "Copy table with headers") -> gr.Button:
366
  return gr.Button(label, size="sm", variant="primary", elem_classes=["copy-btn"])
367
 
368
 
@@ -371,7 +483,7 @@ def _table_heading(text: str) -> gr.HTML:
371
 
372
 
373
  def _copy_ack(_text: str) -> None:
374
- gr.Info("Copied table with headers.", duration=1.2)
375
 
376
 
377
  def _bind_copy(button: gr.Button, source: gr.Textbox) -> None:
@@ -390,64 +502,35 @@ def _analysis_metrics_markdown(result) -> str:
390
 
391
 
392
  def _intervention_metrics_markdown(result) -> str:
393
- drift = f"Execution-context null drift JS: **{result.execution_drift_js:.2e}**"
394
  if result.execution_drift_mean_logprob is not None:
395
- drift += f" · mean log p/token: **{result.execution_drift_mean_logprob:+.2e}**"
396
 
397
- target = (
398
- "No target continuation supplied. The causal comparison therefore uses only "
399
- "next-token Jensen-Shannon divergence."
400
- )
 
401
  if result.baseline_sequence_logprob is not None:
402
  tokens = " ".join(repr(token) for token in result.target_tokens)
403
- target = (
404
- f"**Target continuation:** {result.target_token_count} token(s): {tokens} \n"
405
- f"Sequence log p — baseline: **{result.baseline_sequence_logprob:.4f}** · "
406
- f"SAE edit: **{result.modified_sequence_logprob:.4f}** · "
407
- f"random-ensemble mean: **{result.random_sequence_logprob:.4f}** \n"
408
- f"SAE Δ sequence log p: **{result.sequence_logprob_delta:+.4f}** · "
409
- f"random signed mean Δ: **{result.random_sequence_logprob_delta:+.4f}** \n"
410
- f"SAE Δ mean log p/token: **{result.mean_logprob_delta:+.4f}** · "
411
- f"random mean |Δ|: **{result.random_abs_mean_logprob_delta:.4f}** ± **{result.random_mean_logprob_std:.4f}** · "
412
- f"magnitude ratio: **{result.target_specificity_ratio:.2f}×** · "
413
- f"empirical tail p: **{result.target_empirical_p:.3f}**"
414
- )
415
- inactive = ""
416
  if abs(result.feature_activation) < 1e-12:
417
- inactive = (
418
- " \n**Inactive feature:** ablate/scale has zero coefficient delta here; "
419
- "use `inject` only when you intentionally want a decoder-direction steering test."
420
- )
421
- return (
422
- f"Original feature activation: **{result.feature_activation:.4f}** · "
423
- f"Δ coefficient: **{result.delta_activation:+.4f}** \n"
424
- f"Perturbation L2: **{result.perturbation_norm:.4f}** \n"
425
- f"Next-token JS: **{result.js_divergence:.6f}** · "
426
- f"random ensemble ({result.random_control_count}) mean JS: **{result.random_js_divergence:.6f}** "
427
- f"± **{result.random_js_std:.6f}** · ratio: **{result.js_specificity_ratio:.2f}×** · "
428
- f"empirical tail p: **{result.js_empirical_p:.3f}** \n"
429
- f"{drift} \n\n{target}{inactive} \n\n"
430
- "_Greedy text may remain unchanged even when probability-level causal metrics move; "
431
- "deterministic generation changes only after an argmax boundary is crossed._"
432
- )
433
-
434
 
435
  def _dose_metrics_markdown(result) -> str:
436
  tokens = " ".join(repr(token) for token in result.target_tokens)
437
- inactive = ""
438
- if abs(result.feature_activation) < 1e-12:
439
- inactive = " **The feature is inactive here, so multiplicative scaling is flat by construction.**"
440
  return (
441
- f"Feature activation at baseline: **{result.feature_activation:.4f}** · "
442
- f"target continuation: {len(result.target_tokens)} token(s): {tokens}.{inactive} \n\n"
443
- "This panel is **always a scale intervention**: 0× = ablation, 1× = no edit, "
444
- "2× = double the native coefficient. All rows are compared with the **1× row from the same batched "
445
- "execution context**, so 1× is the numerical null reference. \n"
446
- f"Single-forward → batched-null execution drift: mean log p/token "
447
- f"**{result.execution_drift_mean_logprob:+.2e}**, JS **{result.execution_drift_js:.2e}**."
448
  )
449
 
450
-
451
  def _feature_set_metrics_markdown(result) -> str:
452
  tokens = " ".join(repr(token) for token in result.target_tokens)
453
  inactive_count = sum(abs(float(row[1])) < 1e-12 for row in result.feature_rows)
@@ -477,58 +560,35 @@ def _feature_set_metrics_markdown(result) -> str:
477
  def _interaction_metrics_markdown(result) -> str:
478
  tokens = " ".join(repr(token) for token in result.target_tokens)
479
  return (
480
- f"Target continuation: {len(result.target_tokens)} token(s): {tokens} \n"
481
- f"Additive expectation (sum of individual Δ mean log p/token): "
482
- f"**{result.additive_expected_mean_delta:+.4f}** \n"
483
- f"Observed joint Δ mean log p/token: **{result.joint_mean_delta:+.4f}** \n"
484
- f"Interaction excess (joint additive): **{result.interaction_excess_mean_delta:+.4f}** · "
485
- f"normalized: **{result.normalized_interaction:+.3f}** \n"
486
- f"Execution-context null drift: **{result.execution_drift_mean_logprob:+.2e}** mean log p/token. \n\n"
487
- "A non-zero excess indicates **non-additive downstream response under these edits**; it is not, by itself, "
488
- "proof that the SAE features form a direct mechanistic circuit."
489
  )
490
 
491
-
492
  def _paraphrase_metrics_markdown(result) -> str:
493
  return (
494
- "#### Robustness metrics\n"
495
- "**Selected-token comparison** \n"
496
- f"TopK feature-set Jaccard: **{result.topk_jaccard:.3f}** · "
497
- f"sparse activation cosine: **{result.sparse_cosine:.3f}** \n"
498
- f"Shared features among displayed top-{result.top_n}: **{result.shared_top_n}/{result.top_n}** \n\n"
499
- "**Prompt-wide comparison** — max activation of each feature across all prompt tokens \n"
500
- f"Feature-set Jaccard: **{result.promptwide_jaccard:.3f}** · "
501
- f"activation cosine: **{result.promptwide_cosine:.3f}** \n\n"
502
- "The prompt-wide view reduces sensitivity to choosing semantically different final tokens, while the "
503
- "selected-token view remains the stricter local representation test. Neither metric establishes feature semantics."
504
  )
505
 
506
-
507
  def _concept_metrics_markdown(result) -> str:
508
  coverage = f"{result.active_prompt_count}/{result.total_prompt_count}"
509
  if result.leading_concept is None:
510
- leader = (
511
- "The feature was **inactive in every sampled prompt**. No concept leader is reported; "
512
- "increase the sample or inspect a different feature."
513
- )
514
  elif result.leading_ratio is None:
515
- leader = (
516
- f"Highest prompt-wide mean activation: **{result.leading_concept}**; the runner-up mean was zero."
517
- )
518
  else:
519
- leader = (
520
- f"Highest prompt-wide mean activation: **{result.leading_concept}** "
521
- f"(**{result.leading_ratio:.2f}×** the runner-up mean)."
522
- )
523
  return (
524
- f"Feature **{result.feature_id}**, layer **{result.layer}** · "
525
- f"{result.prompts_per_concept} controlled prompts/concept · active in **{coverage}** sampled prompts. \n"
526
- f"{leader} \n\n"
527
- "Each prompt is summarized by the feature's **maximum activation across its non-padding tokens**. "
528
- "This is an exploratory contrast scan, not an automatic semantic label; held-out AUROC/F1 remains the stronger evidence."
529
  )
530
 
531
-
532
  def _trace_metrics_markdown(result) -> str:
533
  if result.max_token_index is None:
534
  peak = "Feature is inactive at every prompt token."
@@ -545,111 +605,70 @@ def _trace_metrics_markdown(result) -> str:
545
 
546
  def _geometry_metrics_markdown(result) -> str:
547
  if result.alignment_ratio > 1.05:
548
- geometry = "The activation-weighted ablation directions are net **aligned**."
549
  elif result.alignment_ratio < 0.95:
550
- geometry = "The activation-weighted ablation directions show net **cancellation**."
551
  else:
552
- geometry = "The joint norm is close to the orthogonal/independent reference."
553
  return (
554
- f"Selected features: **{', '.join(str(x) for x in result.feature_ids)}** · layer **{result.layer}** \n"
555
- f"Mean |decoder cosine|: **{result.mean_abs_decoder_cosine:.3f}** · "
556
- f"max |decoder cosine|: **{result.max_abs_decoder_cosine:.3f}** \n"
557
- f"Joint ablation L2: **{result.joint_ablation_norm:.4f}** · independent-direction reference: "
558
- f"**{result.independent_norm:.4f}** · geometry ratio: **{result.alignment_ratio:.3f}×** \n"
559
- f"{geometry} This diagnoses SAE decoder geometry; it does not by itself establish downstream causal interaction."
560
  )
561
 
562
-
563
  def _contrastive_metrics_markdown(result) -> str:
564
- direction = "toward A" if result.delta_log_odds > 0 else ("toward B" if result.delta_log_odds < 0 else "neither way")
565
  return (
566
- f"Feature **{result.feature_id}** · activation **{result.feature_activation:.4f}** · perturbation L2 "
567
- f"**{result.perturbation_norm:.4f}** \n"
568
- f"Exact-sequence log-odds A−B — baseline: **{result.baseline_log_odds:+.4f}** · "
569
- f"SAE edit: **{result.modified_log_odds:+.4f}** · causal shift: **{result.delta_log_odds:+.4f}** ({direction}). \n"
570
- f"Token-normalized preference shift: **{result.delta_normalized_preference:+.4f}**. \n"
571
- f"Random ensemble ({result.random_control_count}) |Δ log-odds| mean: **{result.random_abs_mean_delta:.4f}** "
572
- f"± **{result.random_delta_std:.4f}** · SAE/random ratio: **{result.specificity_ratio:.2f}×** · "
573
- f"empirical tail p: **{result.empirical_p:.3f}**. \n\n"
574
- "This asks whether the intervention changes the model's **relative preference between two specified continuations**, "
575
- "which is stricter than showing that one continuation's probability moved in isolation."
576
  )
577
 
578
-
579
-
580
-
581
  def _discovery_metrics_markdown(result) -> str:
582
  if not result.candidate_ids:
583
- if result.ranking_mode == "causal_ready":
584
- return (
585
- f"No positively selective **current-token-active** candidates were found for **{result.concept}** "
586
- f"at layer **{result.layer}** in this live batch. Try another token/layer or use Balanced selectivity "
587
- "to discover concept-associated features elsewhere in the prompt."
588
- )
589
- return (
590
- f"No positively selective candidate features were found for **{result.concept}** at layer "
591
- f"**{result.layer}** in this small live batch. Increase the sample or try another layer."
592
- )
593
  ranking = {
594
- "balanced_selectivity": "balanced selectivity × coverage × log-magnitude",
595
- "raw_mean_difference": "raw target-minus-other mean activation",
596
- "causal_ready": "causal-ready evidence at the current Workbench token",
597
  }[result.ranking_mode]
598
- context = (
599
- f"Current Workbench token **{result.current_token_index}** was checked in the same model batch. "
600
- "The default candidate prefers the highest-ranked feature active at that token when one exists."
601
- if result.current_context_available
602
- else "No Workbench prompt was available, so current-prompt compatibility was not evaluated."
603
- )
604
- stability = ""
605
- if result.split_half_jaccard is not None and result.split_half_k is not None:
606
- stability = (
607
- f" \nSplit-half shortlist stability from the same activation batch: "
608
- f"**{result.split_half_shared_count}** shared candidate(s), Jaccard **{result.split_half_jaccard:.3f}**. "
609
- f"This is a small-sample sensitivity diagnostic, not a reliability estimate."
610
  )
611
  if result.resample_replicates and result.resample_mean_support is not None:
612
- stability += (
613
- f" \nDeterministic balanced-resample support ({result.resample_replicates} resamples, same activations): "
614
- f"mean support across displayed candidates **{result.resample_mean_support:.1%}**; "
615
- f"**{result.resample_high_support_count}/{len(result.candidate_ids)}** displayed candidates appeared in at least 75% "
616
- "of resampled shortlists. This is still a live small-sample stability diagnostic, not a confidence interval."
617
  )
618
- return (
619
- f"Concept **{result.concept}** · layer **{result.layer}** · "
620
- f"{result.prompts_per_concept} prompts/concept. \n"
621
- f"Showing **{len(result.candidate_ids)}** candidates ranked by **{ranking}**. \n"
622
- f"Displayed candidates active at the selected Workbench token: "
623
- f"**{result.displayed_current_active_count}/{len(result.candidate_ids)}**. \n"
624
- f"{context}{stability} \n\n"
625
- "This is **candidate discovery**, not a semantic label. The offline held-out AUROC/F1 pipeline remains "
626
- "the standard for a concept-selectivity claim."
627
- )
628
-
629
 
630
  def _candidate_screen_metrics_markdown(result) -> str:
631
  tokens = " ".join(repr(token) for token in result.target_tokens)
632
  if result.rows:
633
  top = result.rows[0]
634
- strongest = (
635
- f"Strongest screened target effect: feature **{int(top[1])}** with "
636
- f"Δ mean log p/token **{float(top[5]):+.4f}** and next-token JS **{float(top[7]):.6f}**."
637
- )
638
  else:
639
- strongest = "No candidate rows were produced."
640
  return (
641
- f"Screened **{result.candidate_count}** candidate feature(s); **{result.active_feature_count}** were active "
642
- f"at the selected Workbench token. Target continuation: {len(result.target_tokens)} token(s): {tokens}. \n"
643
- f"{strongest} \n"
644
- f"Execution-context null drift: mean log p/token **{result.execution_drift_mean_logprob:+.2e}**, "
645
- f"JS **{result.execution_drift_js:.2e}**. \n\n"
646
- "This is a **triage screen**: all ablations share one batched zero-edit reference, but no random-control "
647
- "ensemble is spent here. Take a promising feature into **Workbench → II. Single-feature causal test** for "
648
- "the full 8-direction specificity comparison."
649
  )
650
 
651
-
652
-
653
  def _spearman_rank_corr(left: list[float], right: list[float]) -> float | None:
654
  """Descriptive Spearman correlation with tie-aware average ranks."""
655
  if len(left) != len(right) or len(left) < 2:
@@ -755,9 +774,7 @@ def _candidate_alignment_outputs(
755
  f"**{int(top_target['Feature id'])}** · strongest next-token distribution shift: **{int(top_js['Feature id'])}**. \n"
756
  f"Spearman ρ(candidate score, |target effect|): **{fmt_rho(rho_target)}** · "
757
  f"ρ(candidate score, next-token JS): **{fmt_rho(rho_js)}**. \n\n"
758
- "This is the project's central association-vs-causality diagnostic: a feature can rank highly by concept evidence yet "
759
- "rank weakly by downstream target effect, or vice versa. With only a handful of screened features and no random-control "
760
- "ensemble in triage, these correlations are **descriptive**, not significance claims."
761
  )
762
 
763
  chart = table[["Feature id", "Candidate score", "|Δ mean log p/token|", "Discovery rank", "Target-effect rank", "Next-token JS"]].copy()
@@ -824,27 +841,16 @@ def _controlled_candidate_shortlist(
824
 
825
  def _candidate_specificity_metrics_markdown(result) -> str:
826
  tokens = " ".join(repr(token) for token in result.target_tokens)
 
827
  if result.rows:
828
  top = result.rows[0]
829
- strongest = (
830
- f"Highest target-specificity ratio: feature **{int(top[1])}** at **{float(top[9]):.2f}×** "
831
- f"the random mean |effect| (empirical tail **{float(top[10]):.3f}**)."
832
- )
833
- else:
834
- strongest = "No controlled candidate rows were produced."
835
  return (
836
- f"Compared **{result.candidate_count}** candidate feature(s); **{result.active_feature_count}** were active "
837
- f"at the selected Workbench token. Each candidate uses **{result.random_control_count}** norm-matched random controls. \n"
838
- f"Target continuation: {len(result.target_tokens)} token(s): {tokens}. \n"
839
- f"{strongest} \n"
840
- f"Execution-context null drift: mean log p/token **{result.execution_drift_mean_logprob:+.2e}**, "
841
- f"JS **{result.execution_drift_js:.2e}**. \n\n"
842
- "This is the controlled follow-up to triage: target-specificity compares |Δ mean log p/token| against the "
843
- "candidate's own norm-matched random ensemble; JS-specificity does the same for the local next-token distribution. "
844
- "With eight controls, empirical tail values remain intentionally coarse."
845
  )
846
 
847
-
848
  def _controlled_evidence_patterns(
849
  specificity_table: pd.DataFrame | None,
850
  ) -> tuple[str, pd.DataFrame]:
@@ -901,8 +907,7 @@ def _controlled_evidence_patterns(
901
  )
902
  summary = (
903
  f"Controlled evidence patterns — {descriptions}. \n\n"
904
- "These labels summarize **effect ratios**, not statistical significance. With eight random controls, empirical tail "
905
- "values are coarse; the offline benchmark is the place for stronger uncertainty estimates."
906
  )
907
  return summary, out
908
 
@@ -937,33 +942,27 @@ def _cross_target_shortlist(specificity_table: pd.DataFrame | None, limit: int =
937
  def _cross_target_metrics_markdown(result) -> str:
938
  feature_text = ", ".join(str(feature_id) for feature_id in result.feature_ids)
939
  target_text = ", ".join(repr(target) for target in result.targets)
 
 
 
 
 
 
 
940
  if result.summary_rows:
941
  strongest = result.summary_rows[0]
942
- lead = (
943
- f"Largest screened cross-target effect: feature **{int(strongest[0])}** on **{strongest[1]!r}** "
944
- f"with Δ mean log p/token **{float(strongest[2]):+.4f}**."
945
  )
946
  patterns = "; ".join(f"{int(row[0])}: {row[10]}" for row in result.summary_rows)
947
- profile_text = f" \nHeuristic target-profile patterns — {patterns}."
948
- else:
949
- lead = "No cross-target rows were produced."
950
- profile_text = ""
951
- pairwise_text = ""
952
  if result.pairwise_rows:
953
  top_pair = result.pairwise_rows[0]
954
- pairwise_text = (
955
- f" \nLargest token-normalized pairwise preference shift: feature **{int(top_pair[0])}**, "
956
- f"**{top_pair[1]!r} vs {top_pair[2]!r}**, Δ(A−B) **{float(top_pair[3]):+.4f}**."
957
  )
958
- return (
959
- f"Profiled feature(s) **{feature_text}** across exact continuation(s) {target_text}; "
960
- f"**{result.active_feature_count}/{len(result.feature_ids)}** selected features were active at the Workbench token. \n"
961
- f"{lead}{profile_text}{pairwise_text} \n\n"
962
- "This is a **target-profile screen** using native SAE ablations and a batched no-edit reference for each continuation. "
963
- "The profile labels are descriptive heuristics, and pairwise shifts compare token-normalized target effects; neither spends "
964
- "random controls. Use Controlled candidate specificity for matched-random causal claims."
965
- )
966
-
967
 
968
  def _controlled_alignment_outputs(
969
  discovery_table: pd.DataFrame | None,
@@ -991,9 +990,7 @@ def _controlled_alignment_outputs(
991
  return "", pd.DataFrame(columns=columns), pd.DataFrame()
992
  if discovery_table is None:
993
  return (
994
- "Controlled specificity results are available, but the concept-discovery table is not present in this browser "
995
- "session, so discovery-rank alignment cannot be reconstructed. The controlled evidence-pattern summary above "
996
- "remains valid; rerun discovery only if you specifically want association-rank alignment.",
997
  pd.DataFrame(columns=columns),
998
  pd.DataFrame(),
999
  )
@@ -1003,8 +1000,7 @@ def _controlled_alignment_outputs(
1003
  return "", pd.DataFrame(columns=columns), pd.DataFrame()
1004
  if discovery.empty or "Feature id" not in discovery:
1005
  return (
1006
- "Controlled specificity results are available, but no concept-discovery evidence is currently populated. "
1007
- "The controlled evidence-pattern summary above remains valid without that association table.",
1008
  pd.DataFrame(columns=columns),
1009
  pd.DataFrame(),
1010
  )
@@ -1114,51 +1110,31 @@ def _controlled_alignment_outputs(
1114
 
1115
 
1116
  def _cue_context_metrics_markdown(result) -> str:
1117
- active = ", ".join(
1118
- f"{cue}: {count}/{len(result.stems)} contexts"
1119
- for cue, count in result.cue_active_context_counts.items()
1120
- )
1121
- interpretation = (
1122
- "No tested cue activated the feature."
1123
- if result.dominant_cue is None or result.active_condition_count == 0
1124
- else (
1125
- f"**Cue-dominant pattern:** `{result.dominant_cue}` activates in every tested context while all "
1126
- "other tested cues are inactive. Under this matrix, the evidence is much more consistent with a "
1127
- "lexical/cue-specific response than with a mathematics-specific response."
1128
- if result.dominant_cue_context_count == len(result.stems)
1129
- and result.off_dominant_active_count == 0
1130
- else (
1131
- f"The strongest cue is `{result.dominant_cue}` ({result.dominant_cue_context_count}/{len(result.stems)} "
1132
- "contexts), but other cues or context dependence remain. Treat the pattern as mixed/context-sensitive."
1133
- )
1134
- )
1135
  )
 
 
 
 
 
 
1136
  return (
1137
- f"Feature **{result.feature_id}**, layer **{result.layer}** · active in "
1138
- f"**{result.active_condition_count}/{result.condition_count}** stem × cue conditions. \n"
1139
- f"Cue coverage — {active}. \n\n"
1140
- f"{interpretation} \n\n"
1141
- "This is a controlled diagnostic over the tested stems/cues, not a universal semantic label."
1142
  )
1143
 
1144
-
1145
  def _cue_metrics_markdown(result) -> str:
1146
  return (
1147
- f"Feature **{result.feature_id}**, layer **{result.layer}** · active for "
1148
- f"**{result.active_cue_count}/{result.cue_count}** tested completion cues at the final token. \n"
1149
- "This helps distinguish a concept-associated feature from a lexical/structural completion-cue feature. "
1150
- "It is a controlled diagnostic, not a feature label."
1151
  )
1152
 
1153
-
1154
  def _global_context_markdown(prompt: str, layer: int, result) -> str:
1155
  token = result.tokens[result.token_index] if result.tokens else ""
1156
  short = prompt[:120] + ("…" if len(prompt) > 120 else "")
1157
  return (
1158
- "**Current Workbench context** "
1159
- f"Prompt: `{short}` · layer **{int(layer)}** · token **{result.token_index}** ({token!r}). \n"
1160
- "Panels marked **Uses current Workbench context** inherit this prompt/layer/token. "
1161
- "Feature selectors remain editable inside each experiment."
1162
  )
1163
 
1164
  @gpu(duration=30)
@@ -1836,8 +1812,10 @@ def run_candidate_cross_target_profile(
1836
  table = pd.DataFrame(result.rows, columns=columns)
1837
  chart = pd.DataFrame(
1838
  result.chart_rows,
1839
- columns=["Target continuation", "Feature", "Δ mean log p/token"],
1840
  )
 
 
1841
  summary_columns = [
1842
  "Feature id",
1843
  "Strongest target",
@@ -1865,7 +1843,9 @@ def run_candidate_cross_target_profile(
1865
  pairwise_chart = pairwise_table.copy()
1866
  if not pairwise_chart.empty:
1867
  pairwise_chart["Target pair"] = pairwise_chart["Target A"].astype(str) + " vs " + pairwise_chart["Target B"].astype(str)
1868
- pairwise_chart["Feature"] = pairwise_chart["Feature id"].astype(str)
 
 
1869
  return (
1870
  _cross_target_metrics_markdown(result),
1871
  table,
@@ -1938,10 +1918,7 @@ def use_candidate_feature(candidate_id: str):
1938
  if candidate_id is None or str(candidate_id).strip() == "":
1939
  raise gr.Error("Run concept-guided discovery and choose a candidate first.")
1940
  value = str(int(float(candidate_id)))
1941
- status = (
1942
- f"**Feature {value} loaded.** It is now selected in Single-feature causal test, "
1943
- "Scale dose-response, Contrastive preference, and Feature evidence. You can still change any selector independently."
1944
- )
1945
  return value, value, value, value, status
1946
 
1947
 
@@ -1962,53 +1939,33 @@ def set_mode_help(mode: str):
1962
  with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_width=True) as demo:
1963
  gr.HTML(
1964
  '<header class="hero">'
1965
- '<h1>FeatureLens <span style="font-size:.48em;opacity:.58;font-weight:400">v0.13</span></h1>'
1966
- '<div class="subtitle">Causal Interpretability Workbench</div>'
1967
- '<div class="question">Discover sparse features, test robustness, and separate correlation from causal influence.</div>'
1968
  '</header>'
1969
  )
1970
 
1971
  global_context = gr.Markdown(
1972
- "**Current Workbench context:** none yet. Go to **Workbench → I. Inspect a prompt location**, "
1973
- "set the prompt/layer/token, then click **Inspect sparse features**.",
1974
  elem_classes=["context-card"],
1975
  )
1976
 
1977
- with gr.Tab("Start here"):
1978
  gr.Markdown(
1979
- "## Start here\n"
1980
- "FeatureLens is organized as a short evidence workflow. You do **not** need to know a feature id before "
1981
- "starting: inspect a prompt first, or use **Feature evidence Concept-guided candidate discovery**."
 
 
1982
  )
1983
- with gr.Row(equal_height=True):
1984
- gr.HTML(
1985
- '<div class="start-card"><h3>1. Choose a location</h3>'
1986
- '<p>In <b>Workbench</b>, enter a prompt, choose a residual layer and token, then click '
1987
- '<b>Inspect sparse features</b>. This establishes the shared Workbench context.</p></div>'
1988
- )
1989
- gr.HTML(
1990
- '<div class="start-card"><h3>2. Choose evidence</h3>'
1991
- '<p>Pick a feature in the experiment you want to run. Feature selectors are populated from the latest '
1992
- 'inspection but remain editable. Feature sets use 2–5 features.</p></div>'
1993
- )
1994
- gr.HTML(
1995
- '<div class="start-card"><h3>3. Ask a causal question</h3>'
1996
- '<p>Ablate/scale features, compare against norm-matched random controls, test dose response, '
1997
- 'preference shifts, robustness, geometry, or concept association.</p></div>'
1998
- )
1999
  gr.Markdown(
2000
- "### Plain-language glossary\n"
2001
- "- **Token:** a chunk of text the model processes.\n"
2002
- "- **Residual layer:** a checkpoint in the model's internal computation.\n"
2003
- "- **SAE feature:** one sparse direction used by the autoencoder to represent a residual activation.\n"
2004
- "- **Ablate:** remove that feature's current contribution. **Scale:** multiply it.\n"
2005
- "- **Random control:** an equally large residual edit in an unrelated direction.\n"
2006
- "- **Teacher-forced score:** probability assigned to exact continuation text, even if greedy generation does not change.\n\n"
2007
- "**Tables:** use the top-right focus control or copy-with-headers action. **Plots:** use the top-right focus control for a bounded reading view; PNG exports use descriptive FeatureLens filenames."
2008
  )
2009
 
2010
  with gr.Tab("Workbench"):
2011
- gr.HTML('<div class="section-rule">I. Inspect a prompt location</div>')
2012
  with gr.Row(equal_height=False):
2013
  with gr.Column(scale=5):
2014
  prompt = gr.Textbox(
@@ -2025,7 +1982,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2025
  ["I am not fully certain, but the answer may be"],
2026
  ],
2027
  inputs=[prompt],
2028
- label="Controlled examples",
2029
  )
2030
  with gr.Column(scale=3):
2031
  layer = gr.Dropdown(
@@ -2042,7 +1999,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2042
  top_n = gr.Slider(5, 20, value=12, step=1, label="Displayed active features")
2043
  analyze_btn = gr.Button("Inspect sparse features", variant="primary", elem_classes=["action-btn"])
2044
 
2045
- gr.Markdown("#### Prompt tokens\nThe selected prompt token is outlined more strongly.")
2046
  token_view = gr.HTML(
2047
  '<div class="small-note">Prompt tokens appear here after clicking <b>Inspect sparse features</b>.</div>'
2048
  )
@@ -2075,18 +2032,18 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2075
  height=330,
2076
  )
2077
 
2078
- gr.HTML('<div class="section-rule">II. Single-feature causal test</div>')
2079
  gr.Markdown(
2080
- "The edit is compared with an **8-direction norm-matched random ensemble** in the same batched "
2081
- "execution context. If a target continuation is supplied, the complete continuation is scored teacher-forced."
2082
  )
2083
  with gr.Row(equal_height=False):
2084
  with gr.Column(scale=2):
2085
  feature_id = gr.Dropdown(
2086
  choices=[],
2087
  allow_custom_value=True,
2088
- label="Single feature id",
2089
- info="Populated from the latest inspection. You can choose another active feature or enter any valid id.",
2090
  )
2091
  mode = gr.Dropdown(
2092
  choices=["ablate", "scale", "inject"],
@@ -2099,9 +2056,9 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2099
  label="Coefficient (unused for ablation)",
2100
  )
2101
  target_text = gr.Textbox(
2102
- label="Target continuation (optional)",
2103
  placeholder="e.g. 2x",
2104
- info="Exact text appended to the prompt for teacher-forced scoring.",
2105
  )
2106
  max_new = gr.Slider(
2107
  4,
@@ -2110,7 +2067,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2110
  step=1,
2111
  label="Greedy generation length",
2112
  )
2113
- intervene_btn = gr.Button("Run single-feature causal test", variant="primary", elem_classes=["action-btn"])
2114
  intervention_metrics = gr.Markdown()
2115
  with gr.Column(scale=3):
2116
  with gr.Row():
@@ -2137,28 +2094,27 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2137
  target_token_tsv = gr.Textbox(visible="hidden")
2138
  target_token_copy = _copy_button()
2139
 
2140
- gr.HTML('<div class="section-rule">III. Single-feature scale dose-response</div>')
2141
  with gr.Group():
2142
- gr.Markdown("**Uses current Workbench prompt / layer / token.** Choose the feature for this experiment below.")
2143
  with gr.Row(equal_height=True):
2144
  dose_feature_id = gr.Dropdown(
2145
  choices=[],
2146
  allow_custom_value=True,
2147
  label="Dose-response feature id",
2148
- info="Populated from Workbench inspection, or enter any valid feature id directly.",
2149
  scale=2,
2150
  )
2151
  dose_target_text = gr.Textbox(
2152
  label="Dose-response target continuation",
2153
  value="2x",
2154
- info="Exact continuation scored across all dose-response conditions.",
2155
  scale=2,
2156
  )
2157
  gr.Markdown(
2158
- "0× = ablation,= numerical no-edit reference, and = double the native coefficient. "
2159
- "The six scale conditions are evaluated together against the 1× row."
2160
  )
2161
- dose_btn = gr.Button("Run scale dose-response", variant="primary", elem_classes=["action-btn"])
2162
  dose_metrics = gr.Markdown()
2163
  with gr.Row(equal_height=False):
2164
  with gr.Column(scale=3):
@@ -2185,19 +2141,17 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2185
  height=330,
2186
  )
2187
 
2188
- gr.HTML('<div class="section-rule">IV. Contrastive continuation preference</div>')
2189
  with gr.Group():
2190
- gr.Markdown("**Uses current Workbench prompt / layer / token.** Select the feature explicitly below.")
2191
  contrastive_feature_id = gr.Dropdown(
2192
  choices=[],
2193
  allow_custom_value=True,
2194
- label="Contrastive feature id",
2195
- info="Populated from the latest Workbench inspection; editable here.",
2196
  )
2197
  gr.Markdown(
2198
- "Absolute target probability can move because an intervention broadly perturbs the distribution. "
2199
- "This test asks the stricter question: **does the edit shift preference between two exact continuations?** "
2200
- "Both continuations are teacher-forced and compared with the same 8-direction norm-matched random ensemble."
2201
  )
2202
  with gr.Row(equal_height=True):
2203
  contrastive_a = gr.Textbox(label="Continuation A (preferred)", value="2x", scale=2)
@@ -2216,7 +2170,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2216
  scale=1,
2217
  )
2218
  contrastive_btn = gr.Button(
2219
- "Run contrastive preference test", variant="primary", elem_classes=["action-btn"]
2220
  )
2221
  contrastive_metrics = gr.Markdown()
2222
  with gr.Row(equal_height=False):
@@ -2246,11 +2200,11 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2246
 
2247
  with gr.Tab("Feature sets"):
2248
  gr.Markdown(
2249
- "### Distributed sparse representations\n"
2250
- "Run **Inspect sparse features** in the Workbench first. **Uses current Workbench context:** prompt, layer, "
2251
- "and token are inherited from that inspection. The feature set below is always editable."
2252
  )
2253
- feature_set_location = gr.Markdown("No Workbench inspection has been run yet.")
2254
  feature_set_ids = gr.Dropdown(
2255
  choices=[],
2256
  value=[],
@@ -2260,11 +2214,9 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2260
  label="Feature set",
2261
  )
2262
 
2263
- gr.Markdown("#### Joint feature-set causal test")
2264
  gr.HTML(
2265
- '<div class="instrument-note">Ablation ignores the multiplier. Scale applies one shared multiplier '
2266
- 'to every selected feature before decoder deltas are summed. Additive injection is deliberately omitted '
2267
- 'because one coefficient is not comparable across unrelated directions.</div>'
2268
  )
2269
  with gr.Row(equal_height=True):
2270
  set_mode = gr.Dropdown(
@@ -2285,7 +2237,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2285
  lines=1,
2286
  scale=2,
2287
  )
2288
- set_btn = gr.Button("Run joint feature-set causal test", variant="primary", elem_classes=["action-btn"])
2289
  set_metrics = gr.Markdown()
2290
  with gr.Row(equal_height=False):
2291
  with gr.Column(scale=2):
@@ -2313,8 +2265,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2313
 
2314
  gr.HTML('<div class="section-rule">Set-size sensitivity</div>')
2315
  gr.Markdown(
2316
- "Ablates the **1, 3, and 5 strongest active features**. Each targeted edit is compared with eight "
2317
- "norm-matched random directions, all sharing one batched no-edit baseline."
2318
  )
2319
  set_sweep_target = gr.Textbox(label="Target continuation for set-size sweep", value="2x")
2320
  set_sweep_btn = gr.Button("Run 1/3/5-feature ablation sweep", variant="primary", elem_classes=["action-btn"])
@@ -2347,11 +2299,10 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2347
  height=330,
2348
  )
2349
 
2350
- gr.HTML('<div class="section-rule">Non-additivity / interaction decomposition</div>')
2351
  gr.Markdown(
2352
- "For the selected 2–5 features, FeatureLens batches each **individual ablation** plus the **joint "
2353
- "ablation**. It compares the observed joint effect with the sum of individual effects. This diagnoses "
2354
- "non-additivity without claiming that the features form a direct circuit."
2355
  )
2356
  interaction_target = gr.Textbox(label="Target continuation for interaction test", value="2x")
2357
  interaction_btn = gr.Button("Run individual-vs-joint decomposition", variant="primary", elem_classes=["action-btn"])
@@ -2386,12 +2337,11 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2386
  gr.HTML('<div class="section-rule">Decoder geometry</div>')
2387
  with gr.Group():
2388
  gr.Markdown(
2389
- "Joint causal effects can reflect both **decoder-direction geometry** and downstream non-linearity. "
2390
- "This zero-extra-generation diagnostic measures pairwise decoder cosines and compares the actual "
2391
- "activation-weighted joint-ablation norm with the norm expected for independent directions."
2392
  )
2393
  geometry_btn = gr.Button(
2394
- "Inspect selected-feature geometry", variant="primary", elem_classes=["action-btn"]
2395
  )
2396
  geometry_metrics = gr.Markdown()
2397
  with gr.Row(equal_height=False):
@@ -2420,19 +2370,16 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2420
  height=320,
2421
  )
2422
 
2423
- with gr.Tab("Feature evidence"):
2424
  gr.Markdown(
2425
- "### Feature activation evidence\n"
2426
- "Inspect **where a feature fires inside the current prompt**, then test whether it activates preferentially "
2427
- "across FeatureLens's seven controlled concept groups. The concept scan uses the feature's **maximum "
2428
- "activation across non-padding tokens in each prompt**, avoiding arbitrary final-token comparisons."
2429
  )
2430
- gr.HTML('<div class="section-rule">A. Concept-guided candidate discovery</div>')
2431
  gr.Markdown(
2432
- "Start from a concept instead of an anonymous feature id. The default **Balanced selectivity** ranking "
2433
- "downweights globally high-activation features and favors candidates that are both selective and consistently "
2434
- "active for the target concept. The table also checks whether each candidate is active in the current Workbench "
2435
- "prompt and selected token, so you can tell whether it is immediately usable for a causal test."
2436
  )
2437
  with gr.Row(equal_height=True):
2438
  discovery_concept = gr.Dropdown(
@@ -2455,10 +2402,10 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2455
  scale=2,
2456
  )
2457
  gr.Markdown(
2458
- "**Input source:** controlled concept prompts + the current Workbench prompt/token, evaluated in the same GPU batch.",
2459
- elem_classes=["instrument-note"],
2460
  )
2461
- discovery_btn = gr.Button("Discover concept-associated candidates", variant="primary", elem_classes=["action-btn"])
2462
  discovery_metrics = gr.Markdown()
2463
  with gr.Row(equal_height=False):
2464
  with gr.Column(scale=3):
@@ -2477,22 +2424,20 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2477
  buttons=["fullscreen", "export"], elem_classes=["fl-plot"], height=330
2478
  )
2479
  gr.Markdown(
2480
- "Click a row in **Candidate feature evidence** to place that feature in the selector below. "
2481
- "The selector defaults to the highest-ranked candidate active at the current Workbench token when available.",
2482
  elem_classes=["candidate-help"],
2483
  )
2484
  with gr.Row(equal_height=True):
2485
  discovery_candidate = gr.Dropdown(choices=[], label="Selected candidate feature id", allow_custom_value=True, scale=3)
2486
  use_candidate_btn = gr.Button(
2487
- "Use selected candidate across feature tests", variant="primary", elem_classes=["action-btn"], scale=2
2488
  )
2489
  candidate_use_status = gr.Markdown()
2490
 
2491
- gr.HTML('<div class="section-rule">B. Batched causal candidate triage</div>')
2492
  gr.Markdown(
2493
- "Use discovery candidates as a short list, then ablate several of them in **one batched scoring run**. "
2494
- "This ranks target effects cheaply before you spend a full 8-random-control causal test on a feature. "
2495
- "Uses the current Workbench prompt/token and the discovery layer."
2496
  )
2497
  with gr.Row(equal_height=True):
2498
  candidate_screen_ids = gr.Dropdown(
@@ -2512,7 +2457,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2512
  scale=2,
2513
  )
2514
  candidate_screen_btn = gr.Button(
2515
- "Screen candidate ablations", variant="primary", elem_classes=["action-btn"]
2516
  )
2517
  candidate_screen_metrics = gr.Markdown()
2518
  with gr.Row(equal_height=False):
@@ -2545,15 +2490,13 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2545
  height=330,
2546
  )
2547
  gr.Markdown(
2548
- "Click a row in **Candidate ablation screen** to move that feature into **Selected candidate feature id**, "
2549
- "then use the existing handoff button above if you want it copied into the deeper feature tests.",
2550
  elem_classes=["candidate-help"],
2551
  )
2552
 
2553
  gr.Markdown("#### Association vs causal influence")
2554
  gr.Markdown(
2555
- "Computed from the discovery table and the ablation screen above **no additional GPU call**. "
2556
- "It compares concept-evidence rank with target-effect and distribution-shift ranks for the same shortlist.",
2557
  elem_classes=["candidate-help"],
2558
  )
2559
  candidate_alignment_metrics = gr.Markdown()
@@ -2594,9 +2537,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2594
 
2595
  gr.Markdown("#### Controlled candidate specificity")
2596
  gr.Markdown(
2597
- "Follow the cheap triage with a small random-controlled comparison. FeatureLens automatically proposes a "
2598
- "three-feature shortlist that preserves the disagreement between discovery, target-effect, and distribution-shift "
2599
- "rankings. Each candidate receives its own 8-direction norm-matched random ensemble in one batched forward."
2600
  )
2601
  with gr.Row(equal_height=True):
2602
  candidate_specificity_ids = gr.Dropdown(
@@ -2616,7 +2558,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2616
  scale=2,
2617
  )
2618
  candidate_specificity_btn = gr.Button(
2619
- "Run controlled candidate comparison", variant="primary", elem_classes=["action-btn"]
2620
  )
2621
  candidate_specificity_metrics = gr.Markdown()
2622
  with gr.Row(equal_height=False):
@@ -2654,8 +2596,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2654
 
2655
  gr.Markdown("#### Controlled evidence patterns")
2656
  gr.Markdown(
2657
- "Zero-GPU synthesis of the controlled table above. It distinguishes target-weighted effects from broader "
2658
- "distribution-shift effects without pretending that eight-control tail values are conventional significance tests.",
2659
  elem_classes=["candidate-help"],
2660
  )
2661
  controlled_pattern_metrics = gr.Markdown()
@@ -2674,8 +2615,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2674
 
2675
  gr.Markdown("#### Association vs controlled causality")
2676
  gr.Markdown(
2677
- "Computed from concept discovery and the random-controlled comparison above. This is the stronger live version "
2678
- "of the association-vs-causality diagnostic because each candidate is normalized against its own matched-control ensemble.",
2679
  elem_classes=["candidate-help"],
2680
  )
2681
  controlled_alignment_metrics = gr.Markdown()
@@ -2714,10 +2654,10 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2714
  height=330,
2715
  )
2716
 
2717
- gr.HTML('<div class="section-rule">C. Cross-target causal profile</div>')
2718
  gr.Markdown(
2719
- "Use the controlled shortlist to test whether a native SAE ablation is concentrated on one exact continuation "
2720
- "or moves several alternatives. This is a cheap target-profile screen: no random controls are spent here."
2721
  )
2722
  with gr.Row(equal_height=True):
2723
  cross_target_ids = gr.Dropdown(
@@ -2738,7 +2678,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2738
  scale=2,
2739
  )
2740
  cross_target_btn = gr.Button(
2741
- "Run cross-target causal profile", variant="primary", elem_classes=["action-btn"]
2742
  )
2743
  cross_target_metrics = gr.Markdown()
2744
  with gr.Row(equal_height=False):
@@ -2759,7 +2699,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2759
  cross_target_plot = gr.BarPlot(
2760
  x="Target continuation",
2761
  y="Δ mean log p/token",
2762
- color="Feature",
 
2763
  title="Candidate effect across exact continuations",
2764
  elem_id="plot-cross-target-profile",
2765
  x_title="Target continuation",
@@ -2783,9 +2724,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2783
 
2784
  gr.Markdown("#### Pairwise target preference shifts")
2785
  gr.Markdown(
2786
- "Derived from the same cross-target scores with **no additional model call**. For each feature, "
2787
- "Δ(A−B) = Δ mean log p/token(A) − Δ mean log p/token(B), so positive values mean the ablation shifts "
2788
- "token-normalized preference toward A relative to B."
2789
  )
2790
  with gr.Row(equal_height=False):
2791
  with gr.Column(scale=3):
@@ -2805,7 +2745,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2805
  cross_target_pairwise_plot = gr.BarPlot(
2806
  x="Target pair",
2807
  y="Δ normalized preference A−B",
2808
- color="Feature",
 
2809
  title="Pairwise target preference shifts",
2810
  elem_id="plot-cross-target-pairwise",
2811
  x_title="Target pair",
@@ -2816,10 +2757,8 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2816
  height=330,
2817
  )
2818
 
2819
- gr.HTML('<div class="section-rule">D. Inspect one feature</div>')
2820
- contrast_location = gr.Markdown(
2821
- "**Activation trace uses the current Workbench prompt.** The controlled concept scan below uses its own balanced prompt set."
2822
- )
2823
  with gr.Row(equal_height=True):
2824
  contrast_feature_id = gr.Dropdown(
2825
  choices=[],
@@ -2842,12 +2781,12 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2842
  scale=2,
2843
  )
2844
 
2845
- gr.Markdown("### Activation trace across the current Workbench prompt")
2846
  gr.Markdown(
2847
- "Uses the prompt currently entered in **Workbench** and shows exactly which prompt tokens include the "
2848
- "selected feature in the SAE TopK support."
2849
  )
2850
- trace_btn = gr.Button("Trace feature across prompt tokens", variant="primary", elem_classes=["action-btn"])
2851
  trace_metrics = gr.Markdown()
2852
  with gr.Row(equal_height=False):
2853
  with gr.Column(scale=3):
@@ -2875,16 +2814,15 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2875
  height=320,
2876
  )
2877
 
2878
- gr.HTML('<div class="section-rule">E. Completion-cue sensitivity</div>')
2879
  gr.Markdown(
2880
- "Feature 22632 in the current math example fires only on the final `is` token. This scan tests whether a "
2881
- "selected feature responds to **completion cues themselves** rather than the underlying concept. Enter a stem "
2882
- "and one cue per line; FeatureLens appends each cue and measures the feature at the final token."
2883
  )
2884
  with gr.Row(equal_height=True):
2885
  cue_stem = gr.Textbox(label="Prompt stem", value="The derivative of x squared", lines=2, scale=3)
2886
  cue_text = gr.Textbox(label="Completion cues (one per line)", value="is\n=\n:\nequals\ntherefore", lines=5, scale=2)
2887
- cue_btn = gr.Button("Run completion-cue scan", variant="primary", elem_classes=["action-btn"])
2888
  cue_metrics = gr.Markdown()
2889
  with gr.Row(equal_height=False):
2890
  with gr.Column(scale=3):
@@ -2899,10 +2837,10 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2899
  buttons=["fullscreen", "export"], elem_classes=["fl-plot"], height=320
2900
  )
2901
 
2902
- gr.HTML('<div class="section-rule">F. Cue × context specificity</div>')
2903
  gr.Markdown(
2904
- "The single-stem cue scan can tell you that a feature prefers `is`, but not whether that preference is lexical "
2905
- "or context-sensitive. This matrix crosses several prompt stems with the same completion cues in one batch."
2906
  )
2907
  with gr.Row(equal_height=True):
2908
  cue_context_stems = gr.Textbox(
@@ -2917,7 +2855,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2917
  lines=5,
2918
  scale=2,
2919
  )
2920
- cue_context_btn = gr.Button("Run cue × context scan", variant="primary", elem_classes=["action-btn"])
2921
  cue_context_metrics = gr.Markdown()
2922
  with gr.Row(equal_height=False):
2923
  with gr.Column(scale=3):
@@ -2954,9 +2892,9 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2954
  height=340,
2955
  )
2956
 
2957
- gr.HTML('<div class="section-rule">G. Controlled concept contrast for selected feature</div>')
2958
- gr.Markdown("### Prompt-wide controlled concept contrast")
2959
- contrast_btn = gr.Button("Run controlled concept contrast", variant="primary", elem_classes=["action-btn"])
2960
  contrast_metrics = gr.Markdown()
2961
  with gr.Row(equal_height=False):
2962
  with gr.Column(scale=3):
@@ -2984,12 +2922,11 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
2984
  height=330,
2985
  )
2986
 
2987
- with gr.Tab("Paraphrase robustness"):
2988
  gr.Markdown(
2989
- "### Local and prompt-wide robustness\n"
2990
- "The selected-token comparison is strict but can be misleading if the two chosen tokens play different "
2991
- "semantic roles. FeatureLens reports a **prompt-wide max-pooled feature profile**: for every SAE "
2992
- "feature, take its maximum activation across all prompt tokens, then compare the two sparse profiles."
2993
  )
2994
  with gr.Row():
2995
  para_a = gr.Textbox(
@@ -3011,7 +2948,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
3011
  para_idx_a = gr.Number(value=-1, precision=0, label="Original prompt token index")
3012
  para_idx_b = gr.Number(value=-1, precision=0, label="Paraphrase token index")
3013
  para_top_n = gr.Slider(5, 20, value=12, step=1, label="Displayed active features")
3014
- para_btn = gr.Button("Compare paraphrase representations", variant="primary", elem_classes=["action-btn"])
3015
  with gr.Row():
3016
  with gr.Column():
3017
  gr.Markdown("#### Original prompt tokens")
@@ -3046,10 +2983,11 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
3046
  height=330,
3047
  )
3048
 
3049
- with gr.Tab("Layer trajectory"):
3050
  gr.Markdown(
3051
- "### Representation structure across early, middle and late residual streams\n"
3052
- "This is not a cross-layer feature-ID comparison; the SAE dictionaries are layer-specific."
 
3053
  )
3054
  with gr.Row():
3055
  trajectory_prompt = gr.Textbox(
@@ -3097,7 +3035,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
3097
  height=330,
3098
  )
3099
 
3100
- with gr.Tab("Offline study"):
3101
  gr.Markdown(STUDY.overview_markdown())
3102
  gr.Markdown(STUDY.readiness_markdown(), elem_classes=["small-note"])
3103
 
@@ -3163,8 +3101,7 @@ with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_w
3163
  )
3164
  else:
3165
  gr.Markdown(
3166
- "The public study tables and figures appear automatically after the small offline artifacts "
3167
- "are generated and committed. No placeholder scientific numbers are shown."
3168
  )
3169
 
3170
  with gr.Tab("Method"):
 
9
  from featurelens.study import OfflineStudy
10
 
11
  # Restrained, print-inspired palette. The app deliberately avoids saturated dashboard colors.
12
+ INK_TEAL = "#6F8984"
13
+ INK_UMBER = "#8A735D"
14
+ INK_RED = "#8C6A67"
15
+ INK_PLUM = "#786F82"
16
+ INK_STONE = "#82827E"
17
+ INK_BLUEGREY = "#687982"
18
 
19
  STUDY = OfflineStudy()
20
 
21
  CSS = r"""
22
+ /*
23
+ FeatureLens UI system
24
+ ---------------------
25
+ This is an analytical instrument, not a SaaS landing page. The interface uses
26
+ plain surfaces, a restrained accent, strong typographic hierarchy, compact
27
+ forms, and deliberate spacing instead of nested cards, badges, glow, or
28
+ uniform full-width CTAs.
29
+ */
30
  .gradio-container {
31
+ --fl-accent: #6F8984;
32
+ --fl-accent-hover: #607A75;
33
+ --fl-rule: var(--border-color-primary);
34
+ --fl-body: "Segoe UI", "Helvetica Neue", Arial, sans-serif;
35
+ --fl-display: Georgia, Cambria, "Times New Roman", serif;
36
+ --fl-mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
37
+ width: min(95vw, 1500px) !important;
38
+ max-width: 1500px !important;
39
+ margin: 0 auto !important;
40
+ padding: 0 24px 112px !important;
41
+ font-family: var(--fl-body) !important;
42
+ font-size: 15.5px !important;
43
+ line-height: 1.48;
44
  }
45
  .gradio-container input,
46
  .gradio-container textarea,
 
49
  .gradio-container label,
50
  .gradio-container table,
51
  .gradio-container .prose {
52
+ font-family: var(--fl-body) !important;
53
  }
54
+ .gradio-container h1,
55
+ .gradio-container h2,
56
+ .gradio-container h3,
57
+ .gradio-container h4,
58
+ .gradio-container .section-rule,
59
+ .gradio-container .table-heading,
60
+ .gradio-container .hero-title {
61
+ font-family: var(--fl-display) !important;
62
+ }
63
+ .gradio-container p,
64
+ .gradio-container li { font-size: 15.5px; }
65
+ .gradio-container h2 { font-size: 1.68rem; line-height: 1.22; margin-bottom: .45rem; }
66
+ .gradio-container h3 { font-size: 1.36rem; line-height: 1.26; margin-bottom: .4rem; }
67
+ .gradio-container h4 { font-size: 1.14rem; line-height: 1.30; margin-bottom: .35rem; }
68
+
69
+ /* Header: compact, editorial, no product-release chrome. */
70
  .hero {
71
+ padding: 18px 0 14px;
72
+ border-bottom: 1px solid var(--fl-rule);
73
+ margin-bottom: 10px;
74
  }
75
  .hero h1 {
76
  margin: 0;
77
+ font-family: var(--fl-display) !important;
78
+ font-size: 2rem;
79
  font-weight: 600;
80
+ line-height: 1.05;
81
+ letter-spacing: -.015em;
82
+ }
83
+ .hero .subtitle {
84
+ margin-top: 6px;
85
+ max-width: 68ch;
86
+ font-size: .98rem;
87
+ color: var(--body-text-color-subdued);
88
+ }
89
+
90
+ /* Navigation is deliberately flat: text tabs + one active rule. */
91
+ .gradio-container .tabs > .tab-nav,
92
+ .gradio-container [role="tablist"] {
93
+ gap: 2px !important;
94
+ border-bottom: 1px solid var(--fl-rule) !important;
95
  }
96
+ .gradio-container [role="tab"] {
97
+ border-radius: 0 !important;
98
+ border: 0 !important;
99
+ background: transparent !important;
100
+ box-shadow: none !important;
101
+ font-weight: 500 !important;
102
+ padding: 9px 11px !important;
103
+ }
104
+ .gradio-container [role="tab"][aria-selected="true"] {
105
+ color: var(--body-text-color) !important;
106
+ border-bottom: 2px solid var(--fl-accent) !important;
107
+ }
108
+
109
+ /* Current context is a quiet status line, not a side-accent card. */
110
+ .context-card {
111
+ border: 0 !important;
112
+ border-bottom: 1px solid var(--fl-rule) !important;
113
+ background: transparent !important;
114
+ padding: 8px 0 10px !important;
115
+ margin: 0 0 14px !important;
116
+ border-radius: 0 !important;
117
+ color: var(--body-text-color-subdued);
118
+ }
119
+ .context-card p { margin: 0 !important; font-size: .93rem !important; }
120
+
121
+ /* Introductory prose stays narrow even when the analytical canvas is wide. */
122
+ .guide-intro { max-width: 72ch; margin-bottom: 10px; }
123
+
124
+ /* Section rhythm: no roman numerals, small-caps, or decorative top rules. */
125
  .section-rule {
126
+ margin: 28px 0 6px;
127
+ padding: 0;
128
+ border: 0;
129
+ font-variant: normal;
130
+ letter-spacing: 0;
131
+ font-size: 1.34rem;
132
+ font-weight: 600;
133
+ opacity: 1;
134
  }
135
+ .section-note,
136
+ .candidate-help,
137
+ .form-note,
138
+ .graph-note,
139
+ .small-note {
140
+ max-width: 78ch;
141
+ color: var(--body-text-color-subdued) !important;
142
+ opacity: 1 !important;
143
+ font-size: .91rem !important;
144
  }
145
+ .section-note { margin: 0 0 12px !important; }
146
+ .form-note { margin: -2px 0 7px !important; }
147
+ .candidate-help { margin-top: 2px !important; }
148
+
149
+ /* Explanatory callouts use a plain rule instead of card / side-tab styling. */
150
  .instrument-note {
151
+ border: 0 !important;
152
+ border-top: 1px solid var(--fl-rule) !important;
153
+ border-bottom: 1px solid var(--fl-rule) !important;
154
+ border-radius: 0 !important;
155
+ padding: 8px 0 !important;
156
+ background: transparent !important;
157
+ margin: 4px 0 12px !important;
158
+ color: var(--body-text-color-subdued);
159
+ font-size: .92rem;
160
+ }
161
+
162
+ /* Forms stay utilitarian and square-ish. */
163
+ .gradio-container .form,
164
+ .gradio-container .block { border-radius: 2px !important; }
165
+ .gradio-container .group {
166
+ border: 0 !important;
167
+ background: transparent !important;
168
+ box-shadow: none !important;
169
+ padding: 0 !important;
170
+ margin: 0 !important;
171
+ }
172
+ .gradio-container textarea,
173
+ .gradio-container input,
174
+ .gradio-container select { border-radius: 2px !important; font-size: 15px !important; }
175
+ .gradio-container label,
176
+ .gradio-container .label-wrap { font-size: 14.5px !important; font-weight: 500 !important; }
177
+
178
+ /* Primary actions are compact; utilities are quiet and secondary. */
179
+ .gradio-container button {
180
+ border-radius: 2px !important;
181
+ font-size: 14.5px !important;
182
+ font-weight: 600 !important;
183
+ box-shadow: none !important;
184
  }
185
+ .action-btn { width: fit-content !important; max-width: 100% !important; }
186
+ .action-btn button {
187
+ width: auto !important;
188
+ min-height: 36px !important;
189
+ padding: 7px 14px !important;
190
+ background: var(--fl-accent) !important;
191
  color: #fff !important;
192
+ border: 1px solid var(--fl-accent) !important;
193
  }
194
+ .action-btn button:hover {
195
+ background: var(--fl-accent-hover) !important;
196
+ border-color: var(--fl-accent-hover) !important;
 
 
 
 
 
 
 
 
 
 
 
197
  }
198
+ .copy-btn { width: fit-content !important; max-width: 100% !important; margin-top: 3px !important; }
199
+ .copy-btn button {
200
+ width: auto !important;
201
+ min-height: 31px !important;
202
+ padding: 5px 10px !important;
203
+ background: transparent !important;
204
+ color: var(--body-text-color-subdued) !important;
205
+ border: 1px solid var(--fl-rule) !important;
206
+ font-weight: 500 !important;
207
+ }
208
+ .copy-btn button:hover {
209
+ background: var(--background-fill-secondary) !important;
210
+ color: var(--body-text-color) !important;
211
+ }
212
+
213
+ /* Tokens are data, so monospace is appropriate here and nowhere else. */
214
+ .token-wrap { display: flex; flex-wrap: wrap; gap: 5px; padding: 6px 0 10px; line-height: 1.85; }
215
+ .token {
216
  background: var(--background-fill-secondary);
217
+ border: 1px solid var(--fl-rule);
 
218
  border-radius: 2px;
219
+ padding: 2px 6px;
220
+ font-family: var(--fl-mono) !important;
221
+ font-size: 12px;
222
  }
223
+ .token.selected { border-color: var(--fl-accent); outline: 1px solid var(--fl-accent); font-weight: 600; }
224
+ .token sup { opacity: .58; margin-right: 4px; }
225
+
226
+ /* Result tables: normal-weight data, tabular numerics, clear heading. */
227
  .table-heading {
228
+ margin: 2px 0 -32px !important;
229
  padding: 4px 58px 0 0 !important;
230
+ min-height: 32px;
231
  position: relative;
232
  z-index: 3;
233
  pointer-events: none;
234
+ font-size: 1.16rem !important;
235
+ font-weight: 600 !important;
236
  line-height: 1.22 !important;
237
  }
238
+ .result-table table,
239
+ .result-table [role="grid"] {
240
+ font-family: var(--fl-body) !important;
241
+ font-size: 14.25px !important;
242
+ font-variant-numeric: tabular-nums;
 
 
 
 
 
 
243
  }
244
  .result-table table thead th,
245
  .result-table table thead th *,
246
  .result-table [role="columnheader"],
247
  .result-table [role="columnheader"] * {
248
+ font-size: 14.25px !important;
249
+ font-weight: 650 !important;
250
+ line-height: 1.24 !important;
251
+ }
252
+ .result-table table tbody td,
253
+ .result-table [role="gridcell"] {
254
+ font-size: 14.25px !important;
255
+ font-weight: 400 !important;
256
+ line-height: 1.32 !important;
257
  }
258
+ .result-table .label-wrap,
259
+ .result-table [data-testid="block-label"],
260
+ .result-table .block-label,
261
+ .result-table .block-title { display: none !important; }
262
+
263
+ /* Plots use the body face for axes/legends; titles can retain their chart style. */
264
+ .fl-plot svg text { font-family: var(--fl-body) !important; }
265
+
266
+ /* Existing in-place focus behavior is preserved exactly. */
267
  .fl-plot.featurelens-inline-focus,
268
  .result-table.featurelens-inline-focus {
269
  position: relative !important;
270
  z-index: 5000 !important;
271
  background: var(--background-fill-primary) !important;
272
+ border: 1px solid var(--fl-rule) !important;
273
+ box-shadow: 0 10px 28px rgba(0, 0, 0, .34) !important;
274
+ border-radius: 2px !important;
 
 
 
 
 
 
275
  }
276
+ .fl-plot.featurelens-inline-focus { transform-origin: top left !important; }
277
+ .result-table.featurelens-inline-focus { overflow: visible !important; }
278
+
279
+ .wide-table { width: 100% !important; }
280
+ .bottom-spacer { height: 86px; width: 100%; }
281
+ .tabs, .tabitem { padding-bottom: 22px !important; }
282
+
283
+ /* Keep prose readable instead of spanning the full analytical canvas. */
284
+ .prose p,
285
+ .prose li { max-width: 80ch; }
286
+
287
  @media (max-width: 900px) {
288
  .gradio-container { width: 100% !important; padding-left: 12px !important; padding-right: 12px !important; }
289
+ .action-btn, .copy-btn { width: 100% !important; }
290
+ .action-btn button, .copy-btn button { width: 100% !important; }
291
  }
292
  """
 
293
  THEME = gr.themes.Base(
294
  primary_hue="teal",
295
+ secondary_hue="gray",
296
+ neutral_hue="gray",
297
  radius_size="sm",
298
  )
299
 
 
305
  const oldLabel = button ? button.innerText : null;
306
  const signal = () => {
307
  if (!button) return;
308
+ button.innerText = "Copied";
309
  button.disabled = true;
310
  window.setTimeout(() => {
311
+ button.innerText = oldLabel || "Copy TSV";
312
  button.disabled = false;
313
  }, 1200);
314
  };
 
474
  return frame.to_csv(sep="\t", index=False, lineterminator="\n")
475
 
476
 
477
+ def _copy_button(label: str = "Copy TSV") -> gr.Button:
478
  return gr.Button(label, size="sm", variant="primary", elem_classes=["copy-btn"])
479
 
480
 
 
483
 
484
 
485
  def _copy_ack(_text: str) -> None:
486
+ gr.Info("Copied TSV with headers.", duration=1.0)
487
 
488
 
489
  def _bind_copy(button: gr.Button, source: gr.Textbox) -> None:
 
502
 
503
 
504
  def _intervention_metrics_markdown(result) -> str:
505
+ drift = f"null drift JS **{result.execution_drift_js:.2e}**"
506
  if result.execution_drift_mean_logprob is not None:
507
+ drift += f" · mean log p/token **{result.execution_drift_mean_logprob:+.2e}**"
508
 
509
+ lines = [
510
+ f"Feature activation **{result.feature_activation:.4f}** · Δ coefficient **{result.delta_activation:+.4f}** · perturbation L2 **{result.perturbation_norm:.4f}**",
511
+ f"Next-token JS **{result.js_divergence:.6f}** · random mean **{result.random_js_divergence:.6f} ± {result.random_js_std:.6f}** · specificity **{result.js_specificity_ratio:.2f}×** · tail **{result.js_empirical_p:.3f}**",
512
+ f"Execution context: {drift}",
513
+ ]
514
  if result.baseline_sequence_logprob is not None:
515
  tokens = " ".join(repr(token) for token in result.target_tokens)
516
+ lines.extend([
517
+ f"Target {tokens} · baseline log p **{result.baseline_sequence_logprob:.4f}** · SAE edit **{result.modified_sequence_logprob:.4f}**",
518
+ f"Δ mean log p/token **{result.mean_logprob_delta:+.4f}** · random mean |Δ| **{result.random_abs_mean_logprob_delta:.4f} ± {result.random_mean_logprob_std:.4f}** · specificity **{result.target_specificity_ratio:.2f}×** · tail **{result.target_empirical_p:.3f}**",
519
+ ])
 
 
 
 
 
 
 
 
 
520
  if abs(result.feature_activation) < 1e-12:
521
+ lines.append("_This feature is inactive at the selected token; ablate/scale therefore has zero native coefficient to remove._")
522
+ elif result.baseline_text == result.modified_text:
523
+ lines.append("_Greedy text is unchanged; the probability-level metrics above are more sensitive than deterministic decoding._")
524
+ return " \n".join(lines)
 
 
 
 
 
 
 
 
 
 
 
 
 
525
 
526
  def _dose_metrics_markdown(result) -> str:
527
  tokens = " ".join(repr(token) for token in result.target_tokens)
528
+ inactive = " · inactive at this token" if abs(result.feature_activation) < 1e-12 else ""
 
 
529
  return (
530
+ f"Feature activation **{result.feature_activation:.4f}**{inactive} · target {tokens} \n"
531
+ f"Reference: **1× no edit** · execution drift mean log p/token **{result.execution_drift_mean_logprob:+.2e}** · JS **{result.execution_drift_js:.2e}**"
 
 
 
 
 
532
  )
533
 
 
534
  def _feature_set_metrics_markdown(result) -> str:
535
  tokens = " ".join(repr(token) for token in result.target_tokens)
536
  inactive_count = sum(abs(float(row[1])) < 1e-12 for row in result.feature_rows)
 
560
  def _interaction_metrics_markdown(result) -> str:
561
  tokens = " ".join(repr(token) for token in result.target_tokens)
562
  return (
563
+ f"Target {tokens} · additive expectation **{result.additive_expected_mean_delta:+.4f}** · "
564
+ f"joint effect **{result.joint_mean_delta:+.4f}** · interaction excess **{result.interaction_excess_mean_delta:+.4f}** "
565
+ f"(normalized **{result.normalized_interaction:+.3f}**) \n"
566
+ f"Execution drift **{result.execution_drift_mean_logprob:+.2e}** mean log p/token. "
567
+ "_Non-additivity is downstream interaction evidence, not a circuit claim._"
 
 
 
 
568
  )
569
 
 
570
  def _paraphrase_metrics_markdown(result) -> str:
571
  return (
572
+ "**Selected token** · "
573
+ f"Jaccard **{result.topk_jaccard:.3f}** · sparse cosine **{result.sparse_cosine:.3f}** · "
574
+ f"shared displayed features **{result.shared_top_n}/{result.top_n}** \n"
575
+ "**Prompt-wide max pool** · "
576
+ f"Jaccard **{result.promptwide_jaccard:.3f}** · cosine **{result.promptwide_cosine:.3f}**"
 
 
 
 
 
577
  )
578
 
 
579
  def _concept_metrics_markdown(result) -> str:
580
  coverage = f"{result.active_prompt_count}/{result.total_prompt_count}"
581
  if result.leading_concept is None:
582
+ leader = "inactive in every sampled prompt"
 
 
 
583
  elif result.leading_ratio is None:
584
+ leader = f"highest mean: **{result.leading_concept}** (runner-up mean 0)"
 
 
585
  else:
586
+ leader = f"highest mean: **{result.leading_concept}** (**{result.leading_ratio:.2f}×** runner-up)"
 
 
 
587
  return (
588
+ f"Feature **{result.feature_id}** · layer **{result.layer}** · active in **{coverage}** prompts · {leader}. \n"
589
+ "_Exploratory prompt-wide contrast; use the Study tab for held-out evidence._"
 
 
 
590
  )
591
 
 
592
  def _trace_metrics_markdown(result) -> str:
593
  if result.max_token_index is None:
594
  peak = "Feature is inactive at every prompt token."
 
605
 
606
  def _geometry_metrics_markdown(result) -> str:
607
  if result.alignment_ratio > 1.05:
608
+ geometry = "net aligned"
609
  elif result.alignment_ratio < 0.95:
610
+ geometry = "net cancelling"
611
  else:
612
+ geometry = "near the independent-direction reference"
613
  return (
614
+ f"Features **{', '.join(str(x) for x in result.feature_ids)}** · layer **{result.layer}** · "
615
+ f"mean |cos| **{result.mean_abs_decoder_cosine:.3f}** · max |cos| **{result.max_abs_decoder_cosine:.3f}** \n"
616
+ f"Joint L2 **{result.joint_ablation_norm:.4f}** · independent reference **{result.independent_norm:.4f}** · "
617
+ f"ratio **{result.alignment_ratio:.3f}×** ({geometry})."
 
 
618
  )
619
 
 
620
  def _contrastive_metrics_markdown(result) -> str:
621
+ direction = "toward A" if result.delta_log_odds > 0 else ("toward B" if result.delta_log_odds < 0 else "no shift")
622
  return (
623
+ f"Feature **{result.feature_id}** · activation **{result.feature_activation:.4f}** · perturbation L2 **{result.perturbation_norm:.4f}** \n"
624
+ f"Exact-sequence log-odds A−B: baseline **{result.baseline_log_odds:+.4f}** · edit **{result.modified_log_odds:+.4f}** · "
625
+ f"shift **{result.delta_log_odds:+.4f}** ({direction}) \n"
626
+ f"Token-normalized shift **{result.delta_normalized_preference:+.4f}** · random mean |Δ| **{result.random_abs_mean_delta:.4f} ± {result.random_delta_std:.4f}** · "
627
+ f"specificity **{result.specificity_ratio:.2f}×** · tail **{result.empirical_p:.3f}**"
 
 
 
 
 
628
  )
629
 
 
 
 
630
  def _discovery_metrics_markdown(result) -> str:
631
  if not result.candidate_ids:
632
+ scope = "current-token-active " if result.ranking_mode == "causal_ready" else ""
633
+ return f"No positively selective {scope}candidates found for **{result.concept}** at layer **{result.layer}** in this live batch."
634
+
 
 
 
 
 
 
 
635
  ranking = {
636
+ "balanced_selectivity": "balanced selectivity",
637
+ "raw_mean_difference": "raw mean difference",
638
+ "causal_ready": "causal-ready evidence",
639
  }[result.ranking_mode]
640
+ lines = [
641
+ f"**{result.concept}** · layer **{result.layer}** · {result.prompts_per_concept} prompts/concept · **{len(result.candidate_ids)}** candidates by {ranking}",
642
+ ]
643
+ if result.current_context_available:
644
+ lines.append(
645
+ f"Current token **{result.current_token_index}** · active candidates **{result.displayed_current_active_count}/{len(result.candidate_ids)}**"
646
+ )
647
+ if result.split_half_jaccard is not None:
648
+ lines.append(
649
+ f"Split-half shortlist · shared **{result.split_half_shared_count}** · Jaccard **{result.split_half_jaccard:.3f}**"
 
 
650
  )
651
  if result.resample_replicates and result.resample_mean_support is not None:
652
+ lines.append(
653
+ f"{result.resample_replicates} resamples · mean shortlist support **{result.resample_mean_support:.1%}** · "
654
+ f"≥75% support **{result.resample_high_support_count}/{len(result.candidate_ids)}** · not a confidence interval"
 
 
655
  )
656
+ lines.append("_Live discovery is exploratory; semantic claims belong to the held-out Study results._")
657
+ return " \n".join(lines)
 
 
 
 
 
 
 
 
 
658
 
659
  def _candidate_screen_metrics_markdown(result) -> str:
660
  tokens = " ".join(repr(token) for token in result.target_tokens)
661
  if result.rows:
662
  top = result.rows[0]
663
+ strongest = f"top feature **{int(top[1])}** · Δ mean log p/token **{float(top[5]):+.4f}** · JS **{float(top[7]):.6f}**"
 
 
 
664
  else:
665
+ strongest = "no candidate rows"
666
  return (
667
+ f"Screened **{result.candidate_count}** candidates · active **{result.active_feature_count}** · target {tokens} · {strongest} \n"
668
+ f"Null drift mean log p/token **{result.execution_drift_mean_logprob:+.2e}** · JS **{result.execution_drift_js:.2e}**. "
669
+ "_This is a triage screen; no random-control ensemble is spent here._"
 
 
 
 
 
670
  )
671
 
 
 
672
  def _spearman_rank_corr(left: list[float], right: list[float]) -> float | None:
673
  """Descriptive Spearman correlation with tie-aware average ranks."""
674
  if len(left) != len(right) or len(left) < 2:
 
774
  f"**{int(top_target['Feature id'])}** · strongest next-token distribution shift: **{int(top_js['Feature id'])}**. \n"
775
  f"Spearman ρ(candidate score, |target effect|): **{fmt_rho(rho_target)}** · "
776
  f"ρ(candidate score, next-token JS): **{fmt_rho(rho_js)}**. \n\n"
777
+ "_descriptive shortlist comparison; triage does not use random controls._"
 
 
778
  )
779
 
780
  chart = table[["Feature id", "Candidate score", "|Δ mean log p/token|", "Discovery rank", "Target-effect rank", "Next-token JS"]].copy()
 
841
 
842
  def _candidate_specificity_metrics_markdown(result) -> str:
843
  tokens = " ".join(repr(token) for token in result.target_tokens)
844
+ strongest = "no controlled rows"
845
  if result.rows:
846
  top = result.rows[0]
847
+ strongest = f"top target specificity **{int(top[1])}** at **{float(top[9]):.2f}×** random mean |effect| · tail **{float(top[10]):.3f}**"
 
 
 
 
 
848
  return (
849
+ f"Compared **{result.candidate_count}** candidates · active **{result.active_feature_count}** · "
850
+ f"**{result.random_control_count}** random controls each · target {tokens} · {strongest} \n"
851
+ f"Null drift mean log p/token **{result.execution_drift_mean_logprob:+.2e}** · JS **{result.execution_drift_js:.2e}**"
 
 
 
 
 
 
852
  )
853
 
 
854
  def _controlled_evidence_patterns(
855
  specificity_table: pd.DataFrame | None,
856
  ) -> tuple[str, pd.DataFrame]:
 
907
  )
908
  summary = (
909
  f"Controlled evidence patterns — {descriptions}. \n\n"
910
+ "_Pattern labels summarize effect ratios, not statistical significance._"
 
911
  )
912
  return summary, out
913
 
 
942
  def _cross_target_metrics_markdown(result) -> str:
943
  feature_text = ", ".join(str(feature_id) for feature_id in result.feature_ids)
944
  target_text = ", ".join(repr(target) for target in result.targets)
945
+ series_text = " · ".join(
946
+ f"Feature {chr(65 + idx)} = **{feature_id}**" for idx, feature_id in enumerate(result.feature_ids)
947
+ )
948
+ lines = [
949
+ f"Features **{feature_text}** · targets {target_text} · active **{result.active_feature_count}/{len(result.feature_ids)}**",
950
+ series_text,
951
+ ]
952
  if result.summary_rows:
953
  strongest = result.summary_rows[0]
954
+ lines.append(
955
+ f"Largest effect: **{int(strongest[0])}** on **{strongest[1]!r}** · Δ mean log p/token **{float(strongest[2]):+.4f}**"
 
956
  )
957
  patterns = "; ".join(f"{int(row[0])}: {row[10]}" for row in result.summary_rows)
958
+ lines.append(f"Profiles · {patterns}")
 
 
 
 
959
  if result.pairwise_rows:
960
  top_pair = result.pairwise_rows[0]
961
+ lines.append(
962
+ f"Largest pairwise preference shift: **{int(top_pair[0])}** · {top_pair[1]!r} vs {top_pair[2]!r} · **{float(top_pair[3]):+.4f}**"
 
963
  )
964
+ lines.append("_Target-profile screen; matched-random specificity is reported above._")
965
+ return " \n".join(lines)
 
 
 
 
 
 
 
966
 
967
  def _controlled_alignment_outputs(
968
  discovery_table: pd.DataFrame | None,
 
990
  return "", pd.DataFrame(columns=columns), pd.DataFrame()
991
  if discovery_table is None:
992
  return (
993
+ "Controlled results are available, but discovery evidence is not present in this browser session, so rank alignment cannot be reconstructed.",
 
 
994
  pd.DataFrame(columns=columns),
995
  pd.DataFrame(),
996
  )
 
1000
  return "", pd.DataFrame(columns=columns), pd.DataFrame()
1001
  if discovery.empty or "Feature id" not in discovery:
1002
  return (
1003
+ "Controlled results are available, but discovery evidence is not currently populated.",
 
1004
  pd.DataFrame(columns=columns),
1005
  pd.DataFrame(),
1006
  )
 
1110
 
1111
 
1112
  def _cue_context_metrics_markdown(result) -> str:
1113
+ active = " · ".join(
1114
+ f"{cue} {count}/{len(result.stems)}" for cue, count in result.cue_active_context_counts.items()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1115
  )
1116
+ if result.dominant_cue is None or result.active_condition_count == 0:
1117
+ interpretation = "No tested cue activated the feature."
1118
+ elif result.dominant_cue_context_count == len(result.stems) and result.off_dominant_active_count == 0:
1119
+ interpretation = f"**Cue-dominant pattern:** `{result.dominant_cue}` activates in every tested context; this is more consistent with a lexical/cue-specific response in this matrix."
1120
+ else:
1121
+ interpretation = f"Strongest cue: `{result.dominant_cue}` ({result.dominant_cue_context_count}/{len(result.stems)} contexts); pattern remains context-dependent."
1122
  return (
1123
+ f"Feature **{result.feature_id}** · layer **{result.layer}** · active **{result.active_condition_count}/{result.condition_count}** conditions \n"
1124
+ f"Cue coverage · {active} \n{interpretation}"
 
 
 
1125
  )
1126
 
 
1127
  def _cue_metrics_markdown(result) -> str:
1128
  return (
1129
+ f"Feature **{result.feature_id}** · layer **{result.layer}** · active for "
1130
+ f"**{result.active_cue_count}/{result.cue_count}** tested cues at the final token."
 
 
1131
  )
1132
 
 
1133
  def _global_context_markdown(prompt: str, layer: int, result) -> str:
1134
  token = result.tokens[result.token_index] if result.tokens else ""
1135
  short = prompt[:120] + ("…" if len(prompt) > 120 else "")
1136
  return (
1137
+ f"**Context** · layer **{int(layer)}** · token **{result.token_index}** ({token!r}) · `{short}`"
 
 
 
1138
  )
1139
 
1140
  @gpu(duration=30)
 
1812
  table = pd.DataFrame(result.rows, columns=columns)
1813
  chart = pd.DataFrame(
1814
  result.chart_rows,
1815
+ columns=["Target continuation", "Feature id", "Δ mean log p/token"],
1816
  )
1817
+ series_lookup = {feature_id: f"Feature {chr(65 + idx)}" for idx, feature_id in enumerate(result.feature_ids)}
1818
+ chart["Series"] = chart["Feature id"].map(lambda value: series_lookup.get(int(value), "Feature"))
1819
  summary_columns = [
1820
  "Feature id",
1821
  "Strongest target",
 
1843
  pairwise_chart = pairwise_table.copy()
1844
  if not pairwise_chart.empty:
1845
  pairwise_chart["Target pair"] = pairwise_chart["Target A"].astype(str) + " vs " + pairwise_chart["Target B"].astype(str)
1846
+ pairwise_chart["Series"] = pairwise_chart["Feature id"].map(
1847
+ lambda value: series_lookup.get(int(value), "Feature")
1848
+ )
1849
  return (
1850
  _cross_target_metrics_markdown(result),
1851
  table,
 
1918
  if candidate_id is None or str(candidate_id).strip() == "":
1919
  raise gr.Error("Run concept-guided discovery and choose a candidate first.")
1920
  value = str(int(float(candidate_id)))
1921
+ status = f"Feature {value} loaded for the feature-level experiments."
 
 
 
1922
  return value, value, value, value, status
1923
 
1924
 
 
1939
  with gr.Blocks(title="FeatureLens — Causal Interpretability Workbench", fill_width=True) as demo:
1940
  gr.HTML(
1941
  '<header class="hero">'
1942
+ '<h1>FeatureLens</h1>'
1943
+ '<div class="subtitle">Sparse-feature analysis for Qwen3-1.7B and Qwen-Scope SAEs.</div>'
 
1944
  '</header>'
1945
  )
1946
 
1947
  global_context = gr.Markdown(
1948
+ "**Context:** no prompt location selected yet.",
 
1949
  elem_classes=["context-card"],
1950
  )
1951
 
1952
+ with gr.Tab("Guide"):
1953
  gr.Markdown(
1954
+ "## Working with the app\n"
1955
+ "Start in **Workbench** to choose a prompt, layer, and token. Use **Feature evidence** when you "
1956
+ "want to find candidates rather than start from a feature id. Return to Workbench or **Feature sets** "
1957
+ "for interventions, and use **Paraphrases**, **Layers**, and **Study** for robustness and aggregate evidence.",
1958
+ elem_classes=["guide-intro"],
1959
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1960
  gr.Markdown(
1961
+ "### Evidence, not labels\n"
1962
+ "A high activation only says that a feature is present. Association is evaluated separately from causal effect; "
1963
+ "random-normalized interventions are the stronger live causal check. Stability metrics show how sensitive a result is to wording or sample choice.\n\n"
1964
+ "Dense tables and plots can be focused in place. **Copy TSV** includes the header row."
 
 
 
 
1965
  )
1966
 
1967
  with gr.Tab("Workbench"):
1968
+ gr.HTML('<div class="section-rule">Inspect</div>')
1969
  with gr.Row(equal_height=False):
1970
  with gr.Column(scale=5):
1971
  prompt = gr.Textbox(
 
1982
  ["I am not fully certain, but the answer may be"],
1983
  ],
1984
  inputs=[prompt],
1985
+ label="Examples",
1986
  )
1987
  with gr.Column(scale=3):
1988
  layer = gr.Dropdown(
 
1999
  top_n = gr.Slider(5, 20, value=12, step=1, label="Displayed active features")
2000
  analyze_btn = gr.Button("Inspect sparse features", variant="primary", elem_classes=["action-btn"])
2001
 
2002
+ gr.Markdown("#### Prompt tokens")
2003
  token_view = gr.HTML(
2004
  '<div class="small-note">Prompt tokens appear here after clicking <b>Inspect sparse features</b>.</div>'
2005
  )
 
2032
  height=330,
2033
  )
2034
 
2035
+ gr.HTML('<div class="section-rule">Single-feature intervention</div>')
2036
  gr.Markdown(
2037
+ "Each edit is compared with eight norm-matched random directions. Add a continuation to score the full exact sequence.",
2038
+ elem_classes=["section-note"],
2039
  )
2040
  with gr.Row(equal_height=False):
2041
  with gr.Column(scale=2):
2042
  feature_id = gr.Dropdown(
2043
  choices=[],
2044
  allow_custom_value=True,
2045
+ label="Feature id",
2046
+ info="Choose an active feature or enter any valid id.",
2047
  )
2048
  mode = gr.Dropdown(
2049
  choices=["ablate", "scale", "inject"],
 
2056
  label="Coefficient (unused for ablation)",
2057
  )
2058
  target_text = gr.Textbox(
2059
+ label="Target continuation",
2060
  placeholder="e.g. 2x",
2061
+ info="Leave blank to compare next-token distributions only.",
2062
  )
2063
  max_new = gr.Slider(
2064
  4,
 
2067
  step=1,
2068
  label="Greedy generation length",
2069
  )
2070
+ intervene_btn = gr.Button("Run intervention", variant="primary", elem_classes=["action-btn"])
2071
  intervention_metrics = gr.Markdown()
2072
  with gr.Column(scale=3):
2073
  with gr.Row():
 
2094
  target_token_tsv = gr.Textbox(visible="hidden")
2095
  target_token_copy = _copy_button()
2096
 
2097
+ gr.HTML('<div class="section-rule">Dose response</div>')
2098
  with gr.Group():
 
2099
  with gr.Row(equal_height=True):
2100
  dose_feature_id = gr.Dropdown(
2101
  choices=[],
2102
  allow_custom_value=True,
2103
  label="Dose-response feature id",
2104
+ info="Choose an active feature or enter any valid id.",
2105
  scale=2,
2106
  )
2107
  dose_target_text = gr.Textbox(
2108
  label="Dose-response target continuation",
2109
  value="2x",
2110
+ info="Exact continuation scored at every multiplier.",
2111
  scale=2,
2112
  )
2113
  gr.Markdown(
2114
+ "0× ablates the feature; is the no-edit reference;doubles the native coefficient.",
2115
+ elem_classes=["section-note"],
2116
  )
2117
+ dose_btn = gr.Button("Measure dose response", variant="primary", elem_classes=["action-btn"])
2118
  dose_metrics = gr.Markdown()
2119
  with gr.Row(equal_height=False):
2120
  with gr.Column(scale=3):
 
2141
  height=330,
2142
  )
2143
 
2144
+ gr.HTML('<div class="section-rule">Contrastive preference</div>')
2145
  with gr.Group():
 
2146
  contrastive_feature_id = gr.Dropdown(
2147
  choices=[],
2148
  allow_custom_value=True,
2149
+ label="Feature id",
2150
+ info="Choose an active feature or enter any valid id.",
2151
  )
2152
  gr.Markdown(
2153
+ "Tests whether the edit changes relative preference between two exact continuations, with the same eight-control random baseline.",
2154
+ elem_classes=["section-note"],
 
2155
  )
2156
  with gr.Row(equal_height=True):
2157
  contrastive_a = gr.Textbox(label="Continuation A (preferred)", value="2x", scale=2)
 
2170
  scale=1,
2171
  )
2172
  contrastive_btn = gr.Button(
2173
+ "Compare continuation preference", variant="primary", elem_classes=["action-btn"]
2174
  )
2175
  contrastive_metrics = gr.Markdown()
2176
  with gr.Row(equal_height=False):
 
2200
 
2201
  with gr.Tab("Feature sets"):
2202
  gr.Markdown(
2203
+ "## Feature sets\n"
2204
+ "Jointly edit active SAE features at the current Workbench location.",
2205
+ elem_classes=["guide-intro"],
2206
  )
2207
+ feature_set_location = gr.Markdown("", visible=False)
2208
  feature_set_ids = gr.Dropdown(
2209
  choices=[],
2210
  value=[],
 
2214
  label="Feature set",
2215
  )
2216
 
2217
+ gr.HTML('<div class="section-rule">Joint intervention</div>')
2218
  gr.HTML(
2219
+ '<div class="instrument-note">Ablation removes each selected feature at its native coefficient. Scale applies one shared multiplier before the decoder deltas are summed.</div>'
 
 
2220
  )
2221
  with gr.Row(equal_height=True):
2222
  set_mode = gr.Dropdown(
 
2237
  lines=1,
2238
  scale=2,
2239
  )
2240
+ set_btn = gr.Button("Run joint intervention", variant="primary", elem_classes=["action-btn"])
2241
  set_metrics = gr.Markdown()
2242
  with gr.Row(equal_height=False):
2243
  with gr.Column(scale=2):
 
2265
 
2266
  gr.HTML('<div class="section-rule">Set-size sensitivity</div>')
2267
  gr.Markdown(
2268
+ "Ablate the 1, 3, and 5 strongest active features with matched random controls.",
2269
+ elem_classes=["section-note"],
2270
  )
2271
  set_sweep_target = gr.Textbox(label="Target continuation for set-size sweep", value="2x")
2272
  set_sweep_btn = gr.Button("Run 1/3/5-feature ablation sweep", variant="primary", elem_classes=["action-btn"])
 
2299
  height=330,
2300
  )
2301
 
2302
+ gr.HTML('<div class="section-rule">Interaction decomposition</div>')
2303
  gr.Markdown(
2304
+ "Compare the joint ablation with the sum of individual effects.",
2305
+ elem_classes=["section-note"],
 
2306
  )
2307
  interaction_target = gr.Textbox(label="Target continuation for interaction test", value="2x")
2308
  interaction_btn = gr.Button("Run individual-vs-joint decomposition", variant="primary", elem_classes=["action-btn"])
 
2337
  gr.HTML('<div class="section-rule">Decoder geometry</div>')
2338
  with gr.Group():
2339
  gr.Markdown(
2340
+ "Decoder-vector cosines and joint-edit geometry.",
2341
+ elem_classes=["section-note"],
 
2342
  )
2343
  geometry_btn = gr.Button(
2344
+ "Inspect decoder geometry", variant="primary", elem_classes=["action-btn"]
2345
  )
2346
  geometry_metrics = gr.Markdown()
2347
  with gr.Row(equal_height=False):
 
2370
  height=320,
2371
  )
2372
 
2373
+ with gr.Tab("Features"):
2374
  gr.Markdown(
2375
+ "## Feature evidence\n"
2376
+ "Discover concept-associated candidates, screen causal effects, then inspect individual features in context.",
2377
+ elem_classes=["guide-intro"],
 
2378
  )
2379
+ gr.HTML('<div class="section-rule">Candidate discovery</div>')
2380
  gr.Markdown(
2381
+ "Rank SAE features from the controlled prompt set. **Causal-ready at current token** keeps only candidates active at the selected Workbench location.",
2382
+ elem_classes=["section-note"],
 
 
2383
  )
2384
  with gr.Row(equal_height=True):
2385
  discovery_concept = gr.Dropdown(
 
2402
  scale=2,
2403
  )
2404
  gr.Markdown(
2405
+ "Controlled prompts and the selected Workbench location share one batch.",
2406
+ elem_classes=["section-note"],
2407
  )
2408
+ discovery_btn = gr.Button("Rank candidates", variant="primary", elem_classes=["action-btn"])
2409
  discovery_metrics = gr.Markdown()
2410
  with gr.Row(equal_height=False):
2411
  with gr.Column(scale=3):
 
2424
  buttons=["fullscreen", "export"], elem_classes=["fl-plot"], height=330
2425
  )
2426
  gr.Markdown(
2427
+ "Click a row to select that feature.",
 
2428
  elem_classes=["candidate-help"],
2429
  )
2430
  with gr.Row(equal_height=True):
2431
  discovery_candidate = gr.Dropdown(choices=[], label="Selected candidate feature id", allow_custom_value=True, scale=3)
2432
  use_candidate_btn = gr.Button(
2433
+ "Use selected feature", variant="secondary", elem_classes=["copy-btn"], scale=2
2434
  )
2435
  candidate_use_status = gr.Markdown()
2436
 
2437
+ gr.HTML('<div class="section-rule">Candidate triage</div>')
2438
  gr.Markdown(
2439
+ "Ablate several candidates in one batch before spending random controls on a smaller shortlist.",
2440
+ elem_classes=["section-note"],
 
2441
  )
2442
  with gr.Row(equal_height=True):
2443
  candidate_screen_ids = gr.Dropdown(
 
2457
  scale=2,
2458
  )
2459
  candidate_screen_btn = gr.Button(
2460
+ "Screen ablations", variant="primary", elem_classes=["action-btn"]
2461
  )
2462
  candidate_screen_metrics = gr.Markdown()
2463
  with gr.Row(equal_height=False):
 
2490
  height=330,
2491
  )
2492
  gr.Markdown(
2493
+ "Click a row to select that feature.",
 
2494
  elem_classes=["candidate-help"],
2495
  )
2496
 
2497
  gr.Markdown("#### Association vs causal influence")
2498
  gr.Markdown(
2499
+ "Discovery rank versus target-effect and distribution-shift rank for the same shortlist.",
 
2500
  elem_classes=["candidate-help"],
2501
  )
2502
  candidate_alignment_metrics = gr.Markdown()
 
2537
 
2538
  gr.Markdown("#### Controlled candidate specificity")
2539
  gr.Markdown(
2540
+ "Up to three candidates, each with its own eight-direction norm-matched random ensemble.",
2541
+ elem_classes=["section-note"],
 
2542
  )
2543
  with gr.Row(equal_height=True):
2544
  candidate_specificity_ids = gr.Dropdown(
 
2558
  scale=2,
2559
  )
2560
  candidate_specificity_btn = gr.Button(
2561
+ "Run controlled comparison", variant="primary", elem_classes=["action-btn"]
2562
  )
2563
  candidate_specificity_metrics = gr.Markdown()
2564
  with gr.Row(equal_height=False):
 
2596
 
2597
  gr.Markdown("#### Controlled evidence patterns")
2598
  gr.Markdown(
2599
+ "Compact reading of target versus distributional specificity.",
 
2600
  elem_classes=["candidate-help"],
2601
  )
2602
  controlled_pattern_metrics = gr.Markdown()
 
2615
 
2616
  gr.Markdown("#### Association vs controlled causality")
2617
  gr.Markdown(
2618
+ "Concept evidence versus random-normalized causal specificity for the same candidates.",
 
2619
  elem_classes=["candidate-help"],
2620
  )
2621
  controlled_alignment_metrics = gr.Markdown()
 
2654
  height=330,
2655
  )
2656
 
2657
+ gr.HTML('<div class="section-rule">Cross-target profile</div>')
2658
  gr.Markdown(
2659
+ "Profile the same native ablation across several exact continuations; random controls are not used in this screen.",
2660
+ elem_classes=["section-note"],
2661
  )
2662
  with gr.Row(equal_height=True):
2663
  cross_target_ids = gr.Dropdown(
 
2678
  scale=2,
2679
  )
2680
  cross_target_btn = gr.Button(
2681
+ "Profile target effects", variant="primary", elem_classes=["action-btn"]
2682
  )
2683
  cross_target_metrics = gr.Markdown()
2684
  with gr.Row(equal_height=False):
 
2699
  cross_target_plot = gr.BarPlot(
2700
  x="Target continuation",
2701
  y="Δ mean log p/token",
2702
+ color="Series",
2703
+ color_map={"Feature A": INK_TEAL, "Feature B": INK_UMBER, "Feature C": INK_PLUM},
2704
  title="Candidate effect across exact continuations",
2705
  elem_id="plot-cross-target-profile",
2706
  x_title="Target continuation",
 
2724
 
2725
  gr.Markdown("#### Pairwise target preference shifts")
2726
  gr.Markdown(
2727
+ "From the same target scores. Positive Δ(A−B) shifts normalized preference toward A.",
2728
+ elem_classes=["section-note"],
 
2729
  )
2730
  with gr.Row(equal_height=False):
2731
  with gr.Column(scale=3):
 
2745
  cross_target_pairwise_plot = gr.BarPlot(
2746
  x="Target pair",
2747
  y="Δ normalized preference A−B",
2748
+ color="Series",
2749
+ color_map={"Feature A": INK_TEAL, "Feature B": INK_UMBER, "Feature C": INK_PLUM},
2750
  title="Pairwise target preference shifts",
2751
  elem_id="plot-cross-target-pairwise",
2752
  x_title="Target pair",
 
2757
  height=330,
2758
  )
2759
 
2760
+ gr.HTML('<div class="section-rule">Inspect one feature</div>')
2761
+ contrast_location = gr.Markdown("", visible=False)
 
 
2762
  with gr.Row(equal_height=True):
2763
  contrast_feature_id = gr.Dropdown(
2764
  choices=[],
 
2781
  scale=2,
2782
  )
2783
 
2784
+ gr.Markdown("### Activation trace")
2785
  gr.Markdown(
2786
+ "Activation of the selected feature across prompt tokens.",
2787
+ elem_classes=["section-note"],
2788
  )
2789
+ trace_btn = gr.Button("Trace feature", variant="primary", elem_classes=["action-btn"])
2790
  trace_metrics = gr.Markdown()
2791
  with gr.Row(equal_height=False):
2792
  with gr.Column(scale=3):
 
2814
  height=320,
2815
  )
2816
 
2817
+ gr.HTML('<div class="section-rule">Completion-cue sensitivity</div>')
2818
  gr.Markdown(
2819
+ "Final-token response to alternative completion cues.",
2820
+ elem_classes=["section-note"],
 
2821
  )
2822
  with gr.Row(equal_height=True):
2823
  cue_stem = gr.Textbox(label="Prompt stem", value="The derivative of x squared", lines=2, scale=3)
2824
  cue_text = gr.Textbox(label="Completion cues (one per line)", value="is\n=\n:\nequals\ntherefore", lines=5, scale=2)
2825
+ cue_btn = gr.Button("Compare completion cues", variant="primary", elem_classes=["action-btn"])
2826
  cue_metrics = gr.Markdown()
2827
  with gr.Row(equal_height=False):
2828
  with gr.Column(scale=3):
 
2837
  buttons=["fullscreen", "export"], elem_classes=["fl-plot"], height=320
2838
  )
2839
 
2840
+ gr.HTML('<div class="section-rule">Cue × context specificity</div>')
2841
  gr.Markdown(
2842
+ "Cross the same cues with unrelated prompt stems.",
2843
+ elem_classes=["section-note"],
2844
  )
2845
  with gr.Row(equal_height=True):
2846
  cue_context_stems = gr.Textbox(
 
2855
  lines=5,
2856
  scale=2,
2857
  )
2858
+ cue_context_btn = gr.Button("Run cue × context", variant="primary", elem_classes=["action-btn"])
2859
  cue_context_metrics = gr.Markdown()
2860
  with gr.Row(equal_height=False):
2861
  with gr.Column(scale=3):
 
2892
  height=340,
2893
  )
2894
 
2895
+ gr.HTML('<div class="section-rule">Controlled concept contrast</div>')
2896
+ gr.Markdown("Prompt-wide max activation across the controlled concept set.", elem_classes=["section-note"])
2897
+ contrast_btn = gr.Button("Compare concepts", variant="primary", elem_classes=["action-btn"])
2898
  contrast_metrics = gr.Markdown()
2899
  with gr.Row(equal_height=False):
2900
  with gr.Column(scale=3):
 
2922
  height=330,
2923
  )
2924
 
2925
+ with gr.Tab("Paraphrases"):
2926
  gr.Markdown(
2927
+ "## Paraphrase robustness\n"
2928
+ "Compare selected-token features with prompt-wide max-pooled SAE profiles.",
2929
+ elem_classes=["guide-intro"],
 
2930
  )
2931
  with gr.Row():
2932
  para_a = gr.Textbox(
 
2948
  para_idx_a = gr.Number(value=-1, precision=0, label="Original prompt token index")
2949
  para_idx_b = gr.Number(value=-1, precision=0, label="Paraphrase token index")
2950
  para_top_n = gr.Slider(5, 20, value=12, step=1, label="Displayed active features")
2951
+ para_btn = gr.Button("Compare representations", variant="primary", elem_classes=["action-btn"])
2952
  with gr.Row():
2953
  with gr.Column():
2954
  gr.Markdown("#### Original prompt tokens")
 
2983
  height=330,
2984
  )
2985
 
2986
+ with gr.Tab("Layers"):
2987
  gr.Markdown(
2988
+ "## Layer trajectory\n"
2989
+ "Compare reconstruction and sparse-activation structure across the selected early, middle, and late SAE layers.",
2990
+ elem_classes=["guide-intro"],
2991
  )
2992
  with gr.Row():
2993
  trajectory_prompt = gr.Textbox(
 
3035
  height=330,
3036
  )
3037
 
3038
+ with gr.Tab("Study"):
3039
  gr.Markdown(STUDY.overview_markdown())
3040
  gr.Markdown(STUDY.readiness_markdown(), elem_classes=["small-note"])
3041
 
 
3101
  )
3102
  else:
3103
  gr.Markdown(
3104
+ "Run the offline study to populate measured tables and figures. Until then, this tab stays intentionally empty."
 
3105
  )
3106
 
3107
  with gr.Tab("Method"):
docs/COLAB.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Running the FeatureLens offline study on Google Colab
2
+
3
+ This guide is for the **offline empirical study**, not the public Hugging Face Space. The Space remains the interactive demo; Colab is only used to produce the study artifacts once.
4
+
5
+ The easiest route is the included notebook:
6
+
7
+ `notebooks/FeatureLens_Offline_Study_Colab.ipynb`
8
+
9
+ ## What the run does
10
+
11
+ The full study executes:
12
+
13
+ 1. build/verify the 224-prompt discovery set and 28 causal tasks;
14
+ 2. collect Qwen3 residuals and prompt-wide Qwen-Scope SAE activations at layers 4, 14, and 26;
15
+ 3. fit/evaluate held-out SAE-feature classifiers and the dense residual probe;
16
+ 4. run single-feature random-controlled causal interventions;
17
+ 5. run 1/3/5 feature-set interventions and controls;
18
+ 6. compute candidate-selection stability, study synthesis, figures, and report;
19
+ 7. validate that the publishable artifacts are complete and methodologically compatible.
20
+
21
+ ## Recommended runtime
22
+
23
+ Use an NVIDIA GPU runtime. A 16 GB T4 is the practical baseline; an L4 or A100 gives more headroom. The notebook detects GPU memory and defaults activation collection to batch size 8 below 20 GB VRAM and 16 otherwise.
24
+
25
+ Runtime varies with Colab allocation, downloads, and prompt lengths. For planning, budget roughly **1–2 hours on a T4** for a first complete run, including model/SAE downloads; faster GPUs can be substantially quicker. Treat this as a planning estimate, not a benchmark.
26
+
27
+ ## Persistence model
28
+
29
+ Colab VMs are temporary. The notebook mounts Google Drive and replaces the repo's `artifacts/` directory with a symlink to a Drive-backed directory. This means:
30
+
31
+ - activation artifacts survive a runtime reset;
32
+ - task-level causal and feature-set checkpoints survive a reset;
33
+ - `python -m experiments.run_all --resume` can continue rather than restart completed work;
34
+ - Hugging Face model caches remain on the Colab VM for speed and may need to be downloaded again after a new runtime.
35
+
36
+ The causal and feature-set stages checkpoint after every completed task in v0.15. Completion-marker files are runtime bookkeeping and are excluded from the publishable bundle.
37
+
38
+ ## Notebook configuration
39
+
40
+ At the top of the notebook set:
41
+
42
+ ```python
43
+ REPO_URL = "PASTE_YOUR_GIT_REPO_URL_HERE"
44
+ BRANCH = "main"
45
+ DRIVE_RUN_NAME = "FeatureLens_offline_v015"
46
+ ```
47
+
48
+ `REPO_URL` can be the public Git URL of the Hugging Face Space repository or another Git mirror containing the same FeatureLens source.
49
+
50
+ If your default branch is not `main`, change `BRANCH`.
51
+
52
+ ## If activation collection runs out of memory
53
+
54
+ The notebook chooses a conservative batch size automatically. If CUDA still runs out of memory, rerun the pipeline cell with:
55
+
56
+ ```bash
57
+ python -m experiments.run_all --resume --activation-batch-size 4
58
+ ```
59
+
60
+ Changing the activation batch size changes memory/time trade-offs, not the experiment definition.
61
+
62
+ ## After the run
63
+
64
+ The notebook runs:
65
+
66
+ ```bash
67
+ python -m scripts.validate_artifacts
68
+ ```
69
+
70
+ and creates a small archive containing only the publishable study outputs. It excludes:
71
+
72
+ - `artifacts/activations/`;
73
+ - `.complete` checkpoint markers;
74
+ - model/SAE caches;
75
+ - temporary files.
76
+
77
+ Extract the archive over the local FeatureLens repository so the files land under `artifacts/`, then run the normal local release checks and push. The public **Study** tab will read those measured artifacts automatically.
78
+
79
+ ## Publishable outputs
80
+
81
+ The bundle is expected to contain files such as:
82
+
83
+ - `artifacts/feature_catalog.csv`
84
+ - `artifacts/layer_metrics.csv`
85
+ - `artifacts/stability.csv`
86
+ - `artifacts/selection_stability.csv`
87
+ - `artifacts/causal_results.csv`
88
+ - `artifacts/feature_set_results.csv`
89
+ - `artifacts/study_feature_summary.csv`
90
+ - `artifacts/study_summary.json`
91
+ - `artifacts/summary.json`
92
+ - `artifacts/report.md`
93
+ - `artifacts/figures/*.png`
94
+
95
+ Do not commit the `artifacts/activations/` directory.
docs/OFFLINE_STUDY.md CHANGED
@@ -17,7 +17,7 @@ The collector also saves `features_final_layer{layer}.npz` so local final-token
17
  ## Full pipeline
18
 
19
  ```bash
20
- python experiments/run_all.py
21
  ```
22
 
23
  Stages:
@@ -35,7 +35,7 @@ Stages:
35
  ## Resume after interruption
36
 
37
  ```bash
38
- python experiments/run_all.py --resume
39
  ```
40
 
41
  The runner checks expected stage outputs and skips completed stages. This is intended for preemptible or quota-limited GPU sessions.
@@ -45,7 +45,7 @@ The runner checks expected stage outputs and skips completed stages. This is int
45
  After activation and causal inference artifacts exist:
46
 
47
  ```bash
48
- python experiments/run_analysis_only.py
49
  ```
50
 
51
  This reruns feature evaluation, candidate stability, study synthesis, figures, report generation, and artifact validation without another model forward pass.
@@ -75,3 +75,21 @@ This reruns feature evaluation, candidate stability, study synthesis, figures, r
75
  - JS specificity asks a different question from target specificity: a feature can reshape the local distribution without specifically controlling the chosen target.
76
  - Cross-concept correlations have n=7 and are descriptive.
77
  - Large activation caches should not be committed to the repository.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
  ## Full pipeline
18
 
19
  ```bash
20
+ python -m experiments.run_all
21
  ```
22
 
23
  Stages:
 
35
  ## Resume after interruption
36
 
37
  ```bash
38
+ python -m experiments.run_all --resume
39
  ```
40
 
41
  The runner checks expected stage outputs and skips completed stages. This is intended for preemptible or quota-limited GPU sessions.
 
45
  After activation and causal inference artifacts exist:
46
 
47
  ```bash
48
+ python -m experiments.run_analysis_only
49
  ```
50
 
51
  This reruns feature evaluation, candidate stability, study synthesis, figures, report generation, and artifact validation without another model forward pass.
 
75
  - JS specificity asks a different question from target specificity: a feature can reshape the local distribution without specifically controlling the chosen target.
76
  - Cross-concept correlations have n=7 and are descriptive.
77
  - Large activation caches should not be committed to the repository.
78
+
79
+ ## Colab workflow
80
+
81
+ A ready-to-run Colab notebook is included at:
82
+
83
+ `notebooks/FeatureLens_Offline_Study_Colab.ipynb`
84
+
85
+ See [`docs/COLAB.md`](COLAB.md) for the persistence model and post-run artifact workflow. The notebook mounts Google Drive for `artifacts/`, keeps the Hugging Face model cache on the local Colab VM, selects a conservative activation batch from available VRAM, and runs the pipeline with `--resume`.
86
+
87
+ v0.15 additionally checkpoints `run_causal` and `run_feature_sets` after each completed task. If a runtime ends mid-stage, the next `--resume` run skips complete tasks within that stage rather than repeating the entire causal or feature-set benchmark.
88
+
89
+ The full runner also accepts:
90
+
91
+ ```bash
92
+ python -m experiments.run_all --resume --activation-batch-size 8 --activation-max-length 192
93
+ ```
94
+
95
+ These two activation flags only affect memory/time during activation collection.
docs/VALIDATION.md CHANGED
@@ -1,10 +1,10 @@
1
- # FeatureLens v0.14 validation
2
 
3
- v0.14 changes the **offline study path and Offline study tab**. It does not change the live Qwen/SAE intervention callbacks validated in v0.13, so do not spend ZeroGPU quota rerunning paraphrase, dose-response, controlled specificity, cross-target, layer trajectory, or feature-set regressions.
4
 
5
- ## A. Local release gate
6
 
7
- From the repository root:
8
 
9
  ```bash
10
  python3 -m pytest -q && \
@@ -14,50 +14,70 @@ python3 scripts/ui_smoke.py && \
14
  python3 scripts/release_check.py
15
  ```
16
 
17
- Expected automated test count for v0.14: **74**.
18
 
19
- ## B. HF acceptance — zero GPU calls
 
 
 
 
 
 
 
20
 
21
- After pushing v0.14:
22
 
23
- 1. Open the Space and confirm startup is clean.
24
- 2. Open **Offline study**.
25
- 3. Before real artifacts are committed, verify it says **Offline study not materialized yet** and does not show fabricated tables or metrics.
26
- 4. Confirm the rest of the live tabs render normally. Do not run GPU experiments solely for this release.
27
 
28
- ## C. Offline-study synthetic/software acceptance
29
 
30
- The automated suite covers:
31
 
32
- - prompt-wide max pooling while ignoring padding;
33
- - balanced candidate score behavior;
34
- - random-ensemble pairing by causal task;
35
- - descriptive Spearman guardrails;
36
- - offline-study missing/complete UI state.
37
 
38
- The development integration gate additionally runs the CPU evaluation/stability/study/report stack on synthetic saved activations and validates the resulting artifact schema.
39
 
40
- ## D. Real offline-study acceptance
41
 
42
- When a CUDA run is available:
43
 
44
- ```bash
45
- python experiments/run_all.py --resume
46
- ```
 
47
 
48
- Then:
49
 
50
- ```bash
51
- python -m scripts.validate_artifacts
52
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
53
 
54
- Required checks:
55
 
56
- - activation metadata says `prompt-wide max activation across non-padding tokens`;
57
- - `features_layer4/14/26.npz` exist for prompt-wide concept evidence;
58
- - `features_final_layer4/14/26.npz` exist for local final-token diagnostics;
59
- - `feature_catalog.csv`, `selection_stability.csv`, `causal_results.csv`, `feature_set_results.csv`, `study_feature_summary.csv`, `study_summary.json`, `summary.json`, and `report.md` are non-empty;
60
- - report figures include `association_vs_causality.png` and `candidate_stability.png`;
61
- - no large activation/model files are staged for Git.
62
 
63
- After committing only the small study artifacts, restart the Space and verify **Offline study** automatically displays the measured study summary, selected-feature table, stability table, layer diagnostics, and figures.
 
1
+ # FeatureLens v0.15 validation
2
 
3
+ v0.15 is a **public-design and offline-runner hardening release**. It does not change the already validated live Qwen/SAE inference methods. Do not spend ZeroGPU quota rerunning causal, paraphrase, layer, cue, discovery, or feature-set experiments for this release.
4
 
5
+ ## Local software gate
6
 
7
+ Run from the repository root:
8
 
9
  ```bash
10
  python3 -m pytest -q && \
 
14
  python3 scripts/release_check.py
15
  ```
16
 
17
+ Then validate the Colab notebook JSON:
18
 
19
+ ```bash
20
+ python3 - <<'PY'
21
+ import nbformat
22
+ nb = nbformat.read('notebooks/FeatureLens_Offline_Study_Colab.ipynb', as_version=4)
23
+ nbformat.validate(nb)
24
+ print('Colab notebook: PASS')
25
+ PY
26
+ ```
27
 
28
+ ## Hugging Face acceptance — no GPU calls
29
 
30
+ After pushing, only inspect the rendered interface.
 
 
 
31
 
32
+ ### A. Header and navigation
33
 
34
+ Pass when:
35
 
36
+ - the header shows **FeatureLens** and one factual subtitle;
37
+ - there is no visible release/version badge;
38
+ - tabs read **Guide, Workbench, Feature sets, Features, Paraphrases, Layers, Study, Method**;
39
+ - tabs are visually flat rather than pill/card navigation.
 
40
 
41
+ ### B. Guide
42
 
43
+ Open **Guide**.
44
 
45
+ Pass when:
46
 
47
+ - there is no three-card “step 1 / step 2 / step 3” onboarding grid;
48
+ - the workflow is short prose;
49
+ - headings use the serif display face while controls/body copy use the neutral sans-serif face;
50
+ - no gradients, glow, badge clusters, or decorative cards are visible.
51
 
52
+ ### C. Workbench without running inference
53
 
54
+ Open **Workbench**.
55
+
56
+ Pass when:
57
+
58
+ - experiment sections have a clear typographic hierarchy;
59
+ - related fields sit close together and separate experiments have more breathing room;
60
+ - primary experiment buttons are compact muted-teal actions, not full-width desktop banners;
61
+ - **Copy TSV** is visually secondary;
62
+ - table titles are clearly larger than table body text;
63
+ - no duplicate context cards appear inside the tab—the global **Context** line is the context source of truth.
64
+
65
+ ### D. Features tab without running inference
66
+
67
+ Open **Features**.
68
+
69
+ Pass when:
70
+
71
+ - discovery, triage, controlled comparison, cross-target profile, and feature diagnostics read as sections of one tool rather than nested cards;
72
+ - helper copy is short and does not repeatedly restate “not a semantic label” after every empty result;
73
+ - the page remains usable in both normal desktop width and a narrower browser window.
74
+
75
+ ### E. Study empty state
76
+
77
+ Open **Study** before real artifacts are committed.
78
 
79
+ Pass when it clearly says the offline study is not materialized and does not show fabricated metrics or placeholder result plots.
80
 
81
+ ## Colab runner dry check
 
 
 
 
 
82
 
83
+ Do not run the full model study merely to validate v0.15. Open the included notebook in Colab and execute only the first GPU-detection cell if desired. The full study should be started only when you are ready to produce the actual empirical artifacts.
experiments/run_all.py CHANGED
@@ -17,14 +17,32 @@ def parse_args() -> argparse.Namespace:
17
  action='store_true',
18
  help='Skip stages whose expected outputs already exist.',
19
  )
 
 
 
 
 
 
 
 
 
 
 
 
20
  return parser.parse_args()
21
 
22
 
23
- def run(module: str, *, outputs: list[Path], resume: bool) -> None:
 
 
 
 
 
 
24
  if resume and outputs and all(path.exists() for path in outputs):
25
  print(f'\nSKIP {module}: expected outputs already exist.', flush=True)
26
  return
27
- command = [sys.executable, '-m', module]
28
  print('\n$', ' '.join(command), flush=True)
29
  subprocess.run(command, cwd=ROOT, check=True)
30
 
@@ -47,6 +65,10 @@ def main() -> None:
47
  *[activation_dir / f'features_final_layer{layer}.npz' for layer in SETTINGS.layers],
48
  ],
49
  resume=args.resume,
 
 
 
 
50
  )
51
  run(
52
  'experiments.evaluate_features',
@@ -58,15 +80,22 @@ def main() -> None:
58
  ],
59
  resume=args.resume,
60
  )
 
61
  run(
62
  'experiments.run_causal',
63
- outputs=[artifact_dir / 'causal_results.csv'],
64
  resume=args.resume,
 
65
  )
 
66
  run(
67
  'experiments.run_feature_sets',
68
- outputs=[artifact_dir / 'feature_set_results.csv'],
 
 
 
69
  resume=args.resume,
 
70
  )
71
  run(
72
  'experiments.analyze_stability',
 
17
  action='store_true',
18
  help='Skip stages whose expected outputs already exist.',
19
  )
20
+ parser.add_argument(
21
+ '--activation-batch-size',
22
+ type=int,
23
+ default=16,
24
+ help='Batch size used only by experiments.collect_activations.',
25
+ )
26
+ parser.add_argument(
27
+ '--activation-max-length',
28
+ type=int,
29
+ default=192,
30
+ help='Maximum prompt length used only by experiments.collect_activations.',
31
+ )
32
  return parser.parse_args()
33
 
34
 
35
+ def run(
36
+ module: str,
37
+ *,
38
+ outputs: list[Path],
39
+ resume: bool,
40
+ extra_args: list[str] | None = None,
41
+ ) -> None:
42
  if resume and outputs and all(path.exists() for path in outputs):
43
  print(f'\nSKIP {module}: expected outputs already exist.', flush=True)
44
  return
45
+ command = [sys.executable, '-m', module, *(extra_args or [])]
46
  print('\n$', ' '.join(command), flush=True)
47
  subprocess.run(command, cwd=ROOT, check=True)
48
 
 
65
  *[activation_dir / f'features_final_layer{layer}.npz' for layer in SETTINGS.layers],
66
  ],
67
  resume=args.resume,
68
+ extra_args=[
69
+ '--batch-size', str(args.activation_batch_size),
70
+ '--max-length', str(args.activation_max_length),
71
+ ],
72
  )
73
  run(
74
  'experiments.evaluate_features',
 
80
  ],
81
  resume=args.resume,
82
  )
83
+ causal_output = artifact_dir / 'causal_results.csv'
84
  run(
85
  'experiments.run_causal',
86
+ outputs=[causal_output, causal_output.with_suffix(causal_output.suffix + '.complete')],
87
  resume=args.resume,
88
+ extra_args=['--resume'] if args.resume else None,
89
  )
90
+ feature_set_output = artifact_dir / 'feature_set_results.csv'
91
  run(
92
  'experiments.run_feature_sets',
93
+ outputs=[
94
+ feature_set_output,
95
+ feature_set_output.with_suffix(feature_set_output.suffix + '.complete'),
96
+ ],
97
  resume=args.resume,
98
+ extra_args=['--resume'] if args.resume else None,
99
  )
100
  run(
101
  'experiments.analyze_stability',
experiments/run_causal.py CHANGED
@@ -22,9 +22,39 @@ def parse_args() -> argparse.Namespace:
22
  parser.add_argument('--output', type=Path, default=ARTIFACT_DIR / 'causal_results.csv')
23
  parser.add_argument('--seed', type=int, default=42)
24
  parser.add_argument('--random-controls', type=int, default=8)
 
 
 
 
 
25
  return parser.parse_args()
26
 
27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  def load_selected_features(path: Path) -> dict[str, dict]:
29
  with path.open(newline='', encoding='utf-8') as handle:
30
  rows = list(csv.DictReader(handle))
@@ -136,8 +166,28 @@ def main() -> None:
136
  top_k=SETTINGS.sae_top_k,
137
  )
138
 
139
- results: list[dict] = []
 
 
 
 
 
 
 
 
 
 
 
 
 
140
  for task_idx, task in enumerate(tasks):
 
 
 
 
 
 
 
141
  concept = task['concept']
142
  choice = selected[concept]
143
  layer = int(choice['layer'])
@@ -284,13 +334,11 @@ def main() -> None:
284
  'top1_changed': int(modified_top1 != baseline_top1),
285
  }
286
  )
 
287
  print(f"Causal task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
288
 
289
- args.output.parent.mkdir(parents=True, exist_ok=True)
290
- with args.output.open('w', newline='', encoding='utf-8') as handle:
291
- writer = csv.DictWriter(handle, fieldnames=list(results[0].keys()))
292
- writer.writeheader()
293
- writer.writerows(results)
294
  print(f'Wrote {len(results)} causal intervention rows to {args.output}')
295
 
296
 
 
22
  parser.add_argument('--output', type=Path, default=ARTIFACT_DIR / 'causal_results.csv')
23
  parser.add_argument('--seed', type=int, default=42)
24
  parser.add_argument('--random-controls', type=int, default=8)
25
+ parser.add_argument(
26
+ '--resume',
27
+ action='store_true',
28
+ help='Resume from task-level rows already checkpointed in --output.',
29
+ )
30
  return parser.parse_args()
31
 
32
 
33
+
34
+
35
+ def _completion_marker(path: Path) -> Path:
36
+ return path.with_suffix(path.suffix + '.complete')
37
+
38
+
39
+ def _write_rows_atomic(path: Path, rows: list[dict]) -> None:
40
+ if not rows:
41
+ return
42
+ path.parent.mkdir(parents=True, exist_ok=True)
43
+ temporary = path.with_suffix(path.suffix + '.tmp')
44
+ with temporary.open('w', newline='', encoding='utf-8') as handle:
45
+ writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys()))
46
+ writer.writeheader()
47
+ writer.writerows(rows)
48
+ temporary.replace(path)
49
+
50
+
51
+ def _load_checkpoint_rows(path: Path) -> list[dict]:
52
+ if not path.exists():
53
+ return []
54
+ with path.open(newline='', encoding='utf-8') as handle:
55
+ return list(csv.DictReader(handle))
56
+
57
+
58
  def load_selected_features(path: Path) -> dict[str, dict]:
59
  with path.open(newline='', encoding='utf-8') as handle:
60
  rows = list(csv.DictReader(handle))
 
166
  top_k=SETTINGS.sae_top_k,
167
  )
168
 
169
+ marker = _completion_marker(args.output)
170
+ if args.resume:
171
+ results: list[dict] = _load_checkpoint_rows(args.output)
172
+ else:
173
+ results = []
174
+ args.output.unlink(missing_ok=True)
175
+ marker.unlink(missing_ok=True)
176
+
177
+ expected_rows_per_task = 2 * (1 + int(args.random_controls))
178
+ completed_counts: dict[str, int] = {}
179
+ for row in results:
180
+ task_id = str(row.get('task_id', ''))
181
+ completed_counts[task_id] = completed_counts.get(task_id, 0) + 1
182
+
183
  for task_idx, task in enumerate(tasks):
184
+ task_id = str(task['id'])
185
+ if args.resume and completed_counts.get(task_id, 0) == expected_rows_per_task:
186
+ print(f"SKIP causal task {task_idx + 1}/{len(tasks)}: {task_id}", flush=True)
187
+ continue
188
+ if args.resume and completed_counts.get(task_id, 0):
189
+ results = [row for row in results if str(row.get('task_id', '')) != task_id]
190
+
191
  concept = task['concept']
192
  choice = selected[concept]
193
  layer = int(choice['layer'])
 
334
  'top1_changed': int(modified_top1 != baseline_top1),
335
  }
336
  )
337
+ _write_rows_atomic(args.output, results)
338
  print(f"Causal task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
339
 
340
+ _write_rows_atomic(args.output, results)
341
+ marker.write_text('complete\n', encoding='utf-8')
 
 
 
342
  print(f'Wrote {len(results)} causal intervention rows to {args.output}')
343
 
344
 
experiments/run_feature_sets.py CHANGED
@@ -27,9 +27,39 @@ def parse_args() -> argparse.Namespace:
27
  parser.add_argument('--sizes', type=int, nargs='+', default=[1, 3, 5])
28
  parser.add_argument('--seed', type=int, default=42)
29
  parser.add_argument('--random-controls', type=int, default=8)
 
 
 
 
 
30
  return parser.parse_args()
31
 
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  def hidden_from_output(output):
34
  return output[0] if isinstance(output, tuple) else output
35
 
@@ -123,11 +153,31 @@ def main() -> None:
123
  top_k=SETTINGS.sae_top_k,
124
  )
125
 
126
- results: list[dict] = []
 
 
 
 
 
 
 
 
 
 
 
 
127
  for task_idx, task in enumerate(tasks):
128
  concept = task['concept']
129
  layer = int(selected[concept]['layer'])
130
  candidate_ids = [int(x) for x in selected[concept]['feature_ids']]
 
 
 
 
 
 
 
 
131
  sae = sae_store.get(layer)
132
  prompt_inputs = tokenizer(task['prompt'], return_tensors='pt', truncation=True, max_length=192)
133
  prompt_inputs = {key: value.to(device) for key, value in prompt_inputs.items()}
@@ -152,7 +202,6 @@ def main() -> None:
152
  residual = capture['hidden'][0, prompt_len - 1]
153
  encoding = sae.encode(residual)
154
 
155
- valid_sizes = [size for size in sizes if size <= len(candidate_ids)]
156
  condition_meta: list[tuple[int, str, int, list[int], torch.Tensor]] = []
157
  for size in valid_sizes:
158
  feature_ids = candidate_ids[:size]
@@ -234,13 +283,11 @@ def main() -> None:
234
  ),
235
  }
236
  )
 
237
  print(f"Feature-set task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
238
 
239
- args.output.parent.mkdir(parents=True, exist_ok=True)
240
- with args.output.open('w', newline='', encoding='utf-8') as handle:
241
- writer = csv.DictWriter(handle, fieldnames=list(results[0].keys()))
242
- writer.writeheader()
243
- writer.writerows(results)
244
  print(f'Wrote {len(results)} feature-set rows to {args.output}')
245
 
246
 
 
27
  parser.add_argument('--sizes', type=int, nargs='+', default=[1, 3, 5])
28
  parser.add_argument('--seed', type=int, default=42)
29
  parser.add_argument('--random-controls', type=int, default=8)
30
+ parser.add_argument(
31
+ '--resume',
32
+ action='store_true',
33
+ help='Resume from task-level rows already checkpointed in --output.',
34
+ )
35
  return parser.parse_args()
36
 
37
 
38
+
39
+
40
+ def _completion_marker(path: Path) -> Path:
41
+ return path.with_suffix(path.suffix + '.complete')
42
+
43
+
44
+ def _write_rows_atomic(path: Path, rows: list[dict]) -> None:
45
+ if not rows:
46
+ return
47
+ path.parent.mkdir(parents=True, exist_ok=True)
48
+ temporary = path.with_suffix(path.suffix + '.tmp')
49
+ with temporary.open('w', newline='', encoding='utf-8') as handle:
50
+ writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys()))
51
+ writer.writeheader()
52
+ writer.writerows(rows)
53
+ temporary.replace(path)
54
+
55
+
56
+ def _load_checkpoint_rows(path: Path) -> list[dict]:
57
+ if not path.exists():
58
+ return []
59
+ with path.open(newline='', encoding='utf-8') as handle:
60
+ return list(csv.DictReader(handle))
61
+
62
+
63
  def hidden_from_output(output):
64
  return output[0] if isinstance(output, tuple) else output
65
 
 
153
  top_k=SETTINGS.sae_top_k,
154
  )
155
 
156
+ marker = _completion_marker(args.output)
157
+ if args.resume:
158
+ results: list[dict] = _load_checkpoint_rows(args.output)
159
+ else:
160
+ results = []
161
+ args.output.unlink(missing_ok=True)
162
+ marker.unlink(missing_ok=True)
163
+
164
+ completed_counts: dict[str, int] = {}
165
+ for row in results:
166
+ task_id = str(row.get('task_id', ''))
167
+ completed_counts[task_id] = completed_counts.get(task_id, 0) + 1
168
+
169
  for task_idx, task in enumerate(tasks):
170
  concept = task['concept']
171
  layer = int(selected[concept]['layer'])
172
  candidate_ids = [int(x) for x in selected[concept]['feature_ids']]
173
+ valid_sizes = [size for size in sizes if size <= len(candidate_ids)]
174
+ expected_rows = len(valid_sizes) * (1 + int(args.random_controls))
175
+ task_id = str(task['id'])
176
+ if args.resume and completed_counts.get(task_id, 0) == expected_rows:
177
+ print(f"SKIP feature-set task {task_idx + 1}/{len(tasks)}: {task_id}", flush=True)
178
+ continue
179
+ if args.resume and completed_counts.get(task_id, 0):
180
+ results = [row for row in results if str(row.get('task_id', '')) != task_id]
181
  sae = sae_store.get(layer)
182
  prompt_inputs = tokenizer(task['prompt'], return_tensors='pt', truncation=True, max_length=192)
183
  prompt_inputs = {key: value.to(device) for key, value in prompt_inputs.items()}
 
202
  residual = capture['hidden'][0, prompt_len - 1]
203
  encoding = sae.encode(residual)
204
 
 
205
  condition_meta: list[tuple[int, str, int, list[int], torch.Tensor]] = []
206
  for size in valid_sizes:
207
  feature_ids = candidate_ids[:size]
 
283
  ),
284
  }
285
  )
286
+ _write_rows_atomic(args.output, results)
287
  print(f"Feature-set task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
288
 
289
+ _write_rows_atomic(args.output, results)
290
+ marker.write_text('complete\n', encoding='utf-8')
 
 
 
291
  print(f'Wrote {len(results)} feature-set rows to {args.output}')
292
 
293
 
notebooks/FeatureLens_Offline_Study_Colab.ipynb ADDED
@@ -0,0 +1,295 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "id": "3ca18c70",
6
+ "metadata": {},
7
+ "source": [
8
+ "# FeatureLens offline study\n",
9
+ "\n",
10
+ "This notebook runs the full FeatureLens empirical study on a CUDA runtime while persisting experiment artifacts to Google Drive. It is designed to be resumable after Colab disconnects.\n",
11
+ "\n",
12
+ "**Before running:** choose a GPU runtime in Colab, then execute the cells from top to bottom."
13
+ ]
14
+ },
15
+ {
16
+ "cell_type": "code",
17
+ "execution_count": null,
18
+ "id": "dd012fdd",
19
+ "metadata": {},
20
+ "outputs": [],
21
+ "source": [
22
+ "# 1. Verify that Colab actually assigned a GPU.\n",
23
+ "import subprocess, sys\n",
24
+ "\n",
25
+ "subprocess.run([\"nvidia-smi\"], check=True)\n",
26
+ "\n",
27
+ "try:\n",
28
+ " import torch\n",
29
+ " assert torch.cuda.is_available(), \"CUDA is not available. Change the Colab runtime to a GPU and reconnect.\"\n",
30
+ " props = torch.cuda.get_device_properties(0)\n",
31
+ " gpu_name = torch.cuda.get_device_name(0)\n",
32
+ " gpu_vram_gb = props.total_memory / 1024**3\n",
33
+ " print(f\"\\nGPU: {gpu_name} | VRAM: {gpu_vram_gb:.1f} GB\")\n",
34
+ "except Exception as exc:\n",
35
+ " raise RuntimeError(\"A CUDA GPU runtime is required for the model stages.\") from exc"
36
+ ]
37
+ },
38
+ {
39
+ "cell_type": "code",
40
+ "execution_count": null,
41
+ "id": "bdac04b1",
42
+ "metadata": {},
43
+ "outputs": [],
44
+ "source": [
45
+ "# 2. Mount Google Drive so completed experiment stages survive a runtime reset.\n",
46
+ "from google.colab import drive\n",
47
+ "drive.mount(\"/content/drive\")"
48
+ ]
49
+ },
50
+ {
51
+ "cell_type": "code",
52
+ "execution_count": null,
53
+ "id": "0446d7cd",
54
+ "metadata": {},
55
+ "outputs": [],
56
+ "source": [
57
+ "# 3. Configuration — edit REPO_URL before running this cell.\n",
58
+ "from pathlib import Path\n",
59
+ "\n",
60
+ "REPO_URL = \"PASTE_YOUR_GIT_REPO_URL_HERE\"\n",
61
+ "BRANCH = \"main\"\n",
62
+ "DRIVE_RUN_NAME = \"FeatureLens_offline_v015\"\n",
63
+ "\n",
64
+ "REPO_DIR = Path(\"/content/FeatureLens\")\n",
65
+ "DRIVE_ROOT = Path(\"/content/drive/MyDrive\") / DRIVE_RUN_NAME\n",
66
+ "DRIVE_ARTIFACTS = DRIVE_ROOT / \"artifacts\"\n",
67
+ "LOG_PATH = DRIVE_ROOT / \"offline_study.log\"\n",
68
+ "\n",
69
+ "if REPO_URL.startswith(\"PASTE_\"):\n",
70
+ " raise ValueError(\"Set REPO_URL to your FeatureLens Git repository URL first.\")\n",
71
+ "\n",
72
+ "DRIVE_ARTIFACTS.mkdir(parents=True, exist_ok=True)\n",
73
+ "print(\"Persistent run directory:\", DRIVE_ROOT)"
74
+ ]
75
+ },
76
+ {
77
+ "cell_type": "code",
78
+ "execution_count": null,
79
+ "id": "c305468e",
80
+ "metadata": {},
81
+ "outputs": [],
82
+ "source": [
83
+ "# 4. Clone or refresh the FeatureLens source.\n",
84
+ "import shutil, subprocess\n",
85
+ "\n",
86
+ "if not REPO_DIR.exists():\n",
87
+ " subprocess.run([\"git\", \"clone\", \"--branch\", BRANCH, \"--single-branch\", REPO_URL, str(REPO_DIR)], check=True)\n",
88
+ "else:\n",
89
+ " subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"fetch\", \"origin\", BRANCH], check=True)\n",
90
+ " subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"checkout\", BRANCH], check=True)\n",
91
+ " subprocess.run([\"git\", \"-C\", str(REPO_DIR), \"pull\", \"--ff-only\", \"origin\", BRANCH], check=True)\n",
92
+ "\n",
93
+ "print(subprocess.check_output([\"git\", \"-C\", str(REPO_DIR), \"rev-parse\", \"--short\", \"HEAD\"], text=True).strip())"
94
+ ]
95
+ },
96
+ {
97
+ "cell_type": "code",
98
+ "execution_count": null,
99
+ "id": "e34c567a",
100
+ "metadata": {},
101
+ "outputs": [],
102
+ "source": [
103
+ "# 5. Install the project environment. This can take a few minutes on a fresh runtime.\n",
104
+ "import subprocess, sys\n",
105
+ "subprocess.run([sys.executable, \"-m\", \"pip\", \"install\", \"-q\", \"-r\", str(REPO_DIR / \"requirements.txt\")], check=True)"
106
+ ]
107
+ },
108
+ {
109
+ "cell_type": "code",
110
+ "execution_count": null,
111
+ "id": "501273c0",
112
+ "metadata": {},
113
+ "outputs": [],
114
+ "source": [
115
+ "# 6. Re-check CUDA after dependency installation and choose a conservative activation batch.\n",
116
+ "import os, torch\n",
117
+ "\n",
118
+ "assert torch.cuda.is_available(), \"CUDA disappeared after dependency setup.\"\n",
119
+ "gpu_name = torch.cuda.get_device_name(0)\n",
120
+ "gpu_vram_gb = torch.cuda.get_device_properties(0).total_memory / 1024**3\n",
121
+ "ACTIVATION_BATCH_SIZE = 16 if gpu_vram_gb >= 20 else 8\n",
122
+ "ACTIVATION_MAX_LENGTH = 192\n",
123
+ "\n",
124
+ "# Keep model/SAE downloads on Colab's local disk for speed.\n",
125
+ "os.environ[\"HF_HOME\"] = \"/content/hf_cache\"\n",
126
+ "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
127
+ "\n",
128
+ "print(f\"GPU: {gpu_name} ({gpu_vram_gb:.1f} GB)\")\n",
129
+ "print(f\"Activation batch size: {ACTIVATION_BATCH_SIZE}\")"
130
+ ]
131
+ },
132
+ {
133
+ "cell_type": "code",
134
+ "execution_count": null,
135
+ "id": "41a3a0e8",
136
+ "metadata": {},
137
+ "outputs": [],
138
+ "source": [
139
+ "# 7. Link FeatureLens artifacts to Google Drive.\n",
140
+ "# Existing small repo artifacts (for example README.md) are copied once; the local directory is then replaced by a symlink.\n",
141
+ "import shutil\n",
142
+ "\n",
143
+ "local_artifacts = REPO_DIR / \"artifacts\"\n",
144
+ "if local_artifacts.is_symlink():\n",
145
+ " local_artifacts.unlink()\n",
146
+ "elif local_artifacts.exists():\n",
147
+ " shutil.copytree(local_artifacts, DRIVE_ARTIFACTS, dirs_exist_ok=True)\n",
148
+ " shutil.rmtree(local_artifacts)\n",
149
+ "\n",
150
+ "local_artifacts.symlink_to(DRIVE_ARTIFACTS, target_is_directory=True)\n",
151
+ "print(\"artifacts ->\", local_artifacts.resolve())"
152
+ ]
153
+ },
154
+ {
155
+ "cell_type": "markdown",
156
+ "id": "dbfda9e4",
157
+ "metadata": {},
158
+ "source": [
159
+ "## Run / resume the study\n",
160
+ "\n",
161
+ "The command below is safe to rerun. Completed stages are skipped. The causal and feature-set stages also checkpoint completed tasks, so a disconnect during either stage does not discard earlier tasks from that stage."
162
+ ]
163
+ },
164
+ {
165
+ "cell_type": "code",
166
+ "execution_count": null,
167
+ "id": "3f0bfd41",
168
+ "metadata": {},
169
+ "outputs": [],
170
+ "source": [
171
+ "# 8. Run the full pipeline with live output and a persistent log.\n",
172
+ "import subprocess, sys, time\n",
173
+ "\n",
174
+ "command = [\n",
175
+ " sys.executable, \"-m\", \"experiments.run_all\",\n",
176
+ " \"--resume\",\n",
177
+ " \"--activation-batch-size\", str(ACTIVATION_BATCH_SIZE),\n",
178
+ " \"--activation-max-length\", str(ACTIVATION_MAX_LENGTH),\n",
179
+ "]\n",
180
+ "\n",
181
+ "print(\"$\", \" \".join(command))\n",
182
+ "print(\"Log:\", LOG_PATH)\n",
183
+ "start = time.time()\n",
184
+ "\n",
185
+ "with LOG_PATH.open(\"a\", encoding=\"utf-8\") as log:\n",
186
+ " log.write(\"\\n\\n=== FeatureLens run ===\\n\")\n",
187
+ " log.write(\"$ \" + \" \".join(command) + \"\\n\")\n",
188
+ " process = subprocess.Popen(\n",
189
+ " command, cwd=REPO_DIR, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, bufsize=1\n",
190
+ " )\n",
191
+ " assert process.stdout is not None\n",
192
+ " for line in process.stdout:\n",
193
+ " print(line, end=\"\")\n",
194
+ " log.write(line)\n",
195
+ " log.flush()\n",
196
+ " return_code = process.wait()\n",
197
+ "\n",
198
+ "if return_code != 0:\n",
199
+ " raise RuntimeError(\n",
200
+ " f\"Pipeline exited with code {return_code}. Fix the error, then rerun this cell; --resume will keep completed work.\"\n",
201
+ " )\n",
202
+ "\n",
203
+ "print(f\"\\nCompleted in {(time.time() - start) / 60:.1f} minutes.\")"
204
+ ]
205
+ },
206
+ {
207
+ "cell_type": "code",
208
+ "execution_count": null,
209
+ "id": "645d7ee5",
210
+ "metadata": {},
211
+ "outputs": [],
212
+ "source": [
213
+ "# 9. Validate the measured artifact set.\n",
214
+ "import subprocess, sys\n",
215
+ "subprocess.run([sys.executable, \"-m\", \"scripts.validate_artifacts\"], cwd=REPO_DIR, check=True)"
216
+ ]
217
+ },
218
+ {
219
+ "cell_type": "code",
220
+ "execution_count": null,
221
+ "id": "9e823e4d",
222
+ "metadata": {},
223
+ "outputs": [],
224
+ "source": [
225
+ "# 10. Inspect the study summary and report.\n",
226
+ "from pathlib import Path\n",
227
+ "import json, pandas as pd\n",
228
+ "from IPython.display import display, Markdown\n",
229
+ "\n",
230
+ "summary_path = DRIVE_ARTIFACTS / \"study_summary.json\"\n",
231
+ "study_table_path = DRIVE_ARTIFACTS / \"study_feature_summary.csv\"\n",
232
+ "report_path = DRIVE_ARTIFACTS / \"report.md\"\n",
233
+ "\n",
234
+ "summary = json.loads(summary_path.read_text(encoding=\"utf-8\"))\n",
235
+ "display(summary)\n",
236
+ "display(pd.read_csv(study_table_path))\n",
237
+ "display(Markdown(report_path.read_text(encoding=\"utf-8\")))"
238
+ ]
239
+ },
240
+ {
241
+ "cell_type": "code",
242
+ "execution_count": null,
243
+ "id": "62c96dc6",
244
+ "metadata": {},
245
+ "outputs": [],
246
+ "source": [
247
+ "# 11. Create a small publishable artifact bundle (activation caches and checkpoint markers are excluded).\n",
248
+ "import zipfile\n",
249
+ "\n",
250
+ "PUBLISH_ZIP = DRIVE_ROOT / \"FeatureLens_offline_results.zip\"\n",
251
+ "\n",
252
+ "with zipfile.ZipFile(PUBLISH_ZIP, \"w\", compression=zipfile.ZIP_DEFLATED) as zf:\n",
253
+ " for path in sorted(DRIVE_ARTIFACTS.rglob(\"*\")):\n",
254
+ " if not path.is_file():\n",
255
+ " continue\n",
256
+ " rel = path.relative_to(DRIVE_ARTIFACTS)\n",
257
+ " if rel.parts and rel.parts[0] == \"activations\":\n",
258
+ " continue\n",
259
+ " if path.name.endswith(\".complete\") or path.name.endswith(\".tmp\"):\n",
260
+ " continue\n",
261
+ " zf.write(path, arcname=str(Path(\"artifacts\") / rel))\n",
262
+ "\n",
263
+ "print(\"Publishable bundle:\", PUBLISH_ZIP)\n",
264
+ "print(f\"Size: {PUBLISH_ZIP.stat().st_size / 1024**2:.2f} MiB\")"
265
+ ]
266
+ },
267
+ {
268
+ "cell_type": "markdown",
269
+ "id": "a6634bfe",
270
+ "metadata": {},
271
+ "source": [
272
+ "## After Colab\n",
273
+ "\n",
274
+ "Download `FeatureLens_offline_results.zip` from the Drive run folder. Extract it over your local FeatureLens repository so the files land under `artifacts/`, run the normal release checks locally, inspect the measured report, and only then commit the small study artifacts. Do **not** commit `artifacts/activations/`."
275
+ ]
276
+ }
277
+ ],
278
+ "metadata": {
279
+ "accelerator": "GPU",
280
+ "colab": {
281
+ "name": "FeatureLens Offline Study",
282
+ "provenance": []
283
+ },
284
+ "kernelspec": {
285
+ "display_name": "Python 3",
286
+ "language": "python",
287
+ "name": "python3"
288
+ },
289
+ "language_info": {
290
+ "name": "python"
291
+ }
292
+ },
293
+ "nbformat": 4,
294
+ "nbformat_minor": 5
295
+ }
pyproject.toml CHANGED
@@ -1,6 +1,6 @@
1
  [project]
2
  name = "featurelens"
3
- version = "0.14.0"
4
  description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
5
  requires-python = ">=3.10"
6
 
 
1
  [project]
2
  name = "featurelens"
3
+ version = "0.15.0"
4
  description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
5
  requires-python = ">=3.10"
6
 
research_config.json CHANGED
@@ -90,7 +90,7 @@
90
  "contrastive_continuation_preference_test",
91
  "feature_decoder_geometry"
92
  ],
93
- "concept_candidate_discovery_metric": "balanced exploratory score = selectivity \u00d7 target activation rate \u00d7 log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
94
  "completion_cue_scan": "final-token feature activation after controlled suffix/cue substitution",
95
  "live_features_v0_6": [
96
  "start_here_plain_language_onboarding",
@@ -196,5 +196,14 @@
196
  "resume_safe_full_study_runner",
197
  "cpu_only_analysis_rerun",
198
  "offline_artifact_schema_validation"
 
 
 
 
 
 
 
 
 
199
  ]
200
  }
 
90
  "contrastive_continuation_preference_test",
91
  "feature_decoder_geometry"
92
  ],
93
+ "concept_candidate_discovery_metric": "balanced exploratory score = selectivity × target activation rate × log1p(target mean); causal-ready mode additionally requires current-token activity and log-scales that activation; raw mean-difference remains available as a scale-sensitive comparison",
94
  "completion_cue_scan": "final-token feature activation after controlled suffix/cue substitution",
95
  "live_features_v0_6": [
96
  "start_here_plain_language_onboarding",
 
196
  "resume_safe_full_study_runner",
197
  "cpu_only_analysis_rerun",
198
  "offline_artifact_schema_validation"
199
+ ],
200
+ "ui_and_runner_features_v0_15": [
201
+ "project_design_contract",
202
+ "flat_research_instrument_visual_system",
203
+ "dual_typeface_hierarchy",
204
+ "concise_data_first_result_copy",
205
+ "muted_cross_target_chart_series",
206
+ "colab_offline_runner_notebook",
207
+ "task_level_causal_and_feature_set_resume"
208
  ]
209
  }
scripts/release_check.py CHANGED
@@ -10,6 +10,7 @@ MAX_FILE_SIZE_BYTES = 5_000_000 # 5 MB
10
 
11
  REQUIRED = [
12
  'README.md',
 
13
  'app.py',
14
  'requirements.txt',
15
  'research_config.json',
@@ -29,6 +30,8 @@ REQUIRED = [
29
  'data/causal_tasks.jsonl',
30
  'docs/VALIDATION.md',
31
  'docs/OFFLINE_STUDY.md',
 
 
32
  'scripts/ui_smoke.py',
33
  'tests/test_offline_study.py',
34
  'scripts/validate_artifacts.py',
@@ -254,6 +257,22 @@ def check_config(config: dict) -> None:
254
  'research_config.json offline_features_v0_14 mismatch: '
255
  f'{sorted(actual_offline_v14)}'
256
  )
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
257
  if config.get('offline_selection_resamples') != 128:
258
  raise SystemExit('Offline selection resamples must be 128.')
259
  if 'prompt-wide' not in str(config.get('offline_feature_pooling', '')):
@@ -320,60 +339,29 @@ def check_readme() -> None:
320
  'sdk: gradio',
321
  'sdk_version: "6.24.0"',
322
  'Qwen/Qwen3-1.7B-Base',
323
- 'full-continuation',
324
- 'feature-set',
325
- 'paraphrase',
326
- 'random-control ensemble',
327
- 'batched zero-edit',
328
- 'concept contrast',
329
- 'non-additivity',
330
- 'contrastive',
331
- 'decoder geometry',
332
- 'token activation',
333
- 'concept-guided candidate',
334
- 'completion-cue',
335
- 'cue × context',
336
- 'balanced selectivity',
337
- 'german',
338
- 'start here',
339
- 'causal-ready',
340
- 'cue-dominant',
341
- 'batched causal candidate triage',
342
- 'in-place',
343
- 'gpu',
344
- 'discovery–causality alignment',
345
- 'spearman',
346
- 'rank-shift',
347
- 'no additional gpu',
348
- 'controlled candidate specificity',
349
- 'norm-matched random ensemble',
350
- 'association vs controlled causality',
351
- 'target-specificity ratio',
352
- 'js-specificity ratio',
353
- 'controlled evidence patterns',
354
- 'split-half',
355
- 'cross-target',
356
- 'target-profile',
357
- 'resample shortlist support',
358
- 'pairwise target preference',
359
- 'effect concentration',
360
- 'signed bias',
361
  'prompt-wide',
362
  'offline study',
363
- 'selection stability',
364
- 'run_analysis_only',
365
- '--resume',
366
  'validate_artifacts',
 
 
367
  ]
368
  missing = [value for value in required_strings if value.lower() not in readme.lower()]
369
  if missing:
370
- raise SystemExit(f'README.md is missing required v0.14 content: {missing}')
 
 
 
 
371
 
372
 
373
  def check_pyproject() -> None:
374
  text = (ROOT / 'pyproject.toml').read_text(encoding='utf-8')
375
- if 'version = "0.14.0"' not in text:
376
- raise SystemExit('pyproject.toml must declare version 0.14.0.')
377
 
378
 
379
  def main() -> None:
@@ -391,7 +379,7 @@ def main() -> None:
391
  print(f' layers: {config["layers"]}')
392
  print(f' feature-set sizes: {config["feature_set_sizes"]}')
393
  print(f' random controls: {config["live_random_controls"]}')
394
- print(' release: v0.14.0')
395
 
396
 
397
  if __name__ == '__main__':
 
10
 
11
  REQUIRED = [
12
  'README.md',
13
+ 'DESIGN.md',
14
  'app.py',
15
  'requirements.txt',
16
  'research_config.json',
 
30
  'data/causal_tasks.jsonl',
31
  'docs/VALIDATION.md',
32
  'docs/OFFLINE_STUDY.md',
33
+ 'docs/COLAB.md',
34
+ 'notebooks/FeatureLens_Offline_Study_Colab.ipynb',
35
  'scripts/ui_smoke.py',
36
  'tests/test_offline_study.py',
37
  'scripts/validate_artifacts.py',
 
257
  'research_config.json offline_features_v0_14 mismatch: '
258
  f'{sorted(actual_offline_v14)}'
259
  )
260
+
261
+ required_v15 = {
262
+ 'project_design_contract',
263
+ 'flat_research_instrument_visual_system',
264
+ 'dual_typeface_hierarchy',
265
+ 'concise_data_first_result_copy',
266
+ 'muted_cross_target_chart_series',
267
+ 'colab_offline_runner_notebook',
268
+ 'task_level_causal_and_feature_set_resume',
269
+ }
270
+ actual_v15 = set(config.get('ui_and_runner_features_v0_15', []))
271
+ if actual_v15 != required_v15:
272
+ raise SystemExit(
273
+ 'research_config.json ui_and_runner_features_v0_15 mismatch: '
274
+ f'{sorted(actual_v15)}'
275
+ )
276
  if config.get('offline_selection_resamples') != 128:
277
  raise SystemExit('Offline selection resamples must be 128.')
278
  if 'prompt-wide' not in str(config.get('offline_feature_pooling', '')):
 
339
  'sdk: gradio',
340
  'sdk_version: "6.24.0"',
341
  'Qwen/Qwen3-1.7B-Base',
342
+ 'Qwen-Scope',
343
+ 'random controls',
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
344
  'prompt-wide',
345
  'offline study',
346
+ '-m experiments.run_all --resume',
347
+ '--activation-batch-size',
 
348
  'validate_artifacts',
349
+ 'FeatureLens_Offline_Study_Colab.ipynb',
350
+ 'DESIGN.md',
351
  ]
352
  missing = [value for value in required_strings if value.lower() not in readme.lower()]
353
  if missing:
354
+ raise SystemExit(f'README.md is missing required v0.15 content: {missing}')
355
+
356
+ # Public README should not lead with release-train marketing. Version history belongs in CHANGELOG.
357
+ if '> **v0.' in readme or '## v0.' in readme:
358
+ raise SystemExit('README.md should not contain visible release-announcement/version-history sections.')
359
 
360
 
361
  def check_pyproject() -> None:
362
  text = (ROOT / 'pyproject.toml').read_text(encoding='utf-8')
363
+ if 'version = "0.15.0"' not in text:
364
+ raise SystemExit('pyproject.toml must declare version 0.15.0.')
365
 
366
 
367
  def main() -> None:
 
379
  print(f' layers: {config["layers"]}')
380
  print(f' feature-set sizes: {config["feature_set_sizes"]}')
381
  print(f' random controls: {config["live_random_controls"]}')
382
+ print(' release: v0.15.0')
383
 
384
 
385
  if __name__ == '__main__':
tests/test_design_system.py ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from pathlib import Path
2
+
3
+ ROOT = Path(__file__).resolve().parents[1]
4
+ APP = (ROOT / 'app.py').read_text(encoding='utf-8')
5
+ DESIGN = (ROOT / 'DESIGN.md').read_text(encoding='utf-8')
6
+
7
+
8
+ def test_design_contract_is_present_and_specific():
9
+ assert 'research instrument' in DESIGN.lower()
10
+ assert 'no gradients' in DESIGN.lower()
11
+ assert 'nested-card' in DESIGN.lower() or 'nested cards' in DESIGN.lower()
12
+ assert 'Segoe UI' in DESIGN
13
+ assert 'Georgia' in DESIGN
14
+ assert 'Copy TSV' in DESIGN
15
+
16
+
17
+ def test_public_ui_avoids_common_generated_frontend_tells():
18
+ lowered = APP.lower()
19
+ assert 'linear-gradient' not in lowered
20
+ assert 'radial-gradient' not in lowered
21
+ assert 'backdrop-filter' not in lowered
22
+ assert 'gr.accordion(' not in lowered
23
+ assert 'start here' not in lowered
24
+ assert 'unlock your' not in lowered
25
+ assert 'supercharge' not in lowered
26
+ assert 'all-in-one' not in lowered
27
+ assert '<h1>featurelens</h1>' in lowered
28
+ assert '<h1>featurelens v' not in lowered
29
+
30
+
31
+ def test_ui_has_explicit_type_and_action_roles():
32
+ assert '--fl-body: "Segoe UI"' in APP
33
+ assert '--fl-display: Georgia' in APP
34
+ assert '.action-btn' in APP
35
+ assert '.copy-btn' in APP
36
+ assert 'Copy TSV' in APP
37
+ assert 'with gr.Tab("Features")' in APP
38
+
39
+
40
+ def test_cross_target_charts_use_restrained_fixed_series_palette():
41
+ assert '"Feature A": INK_TEAL' in APP
42
+ assert '"Feature B": INK_UMBER' in APP
43
+ assert '"Feature C": INK_PLUM' in APP
tests/test_offline_resume.py ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import csv
4
+ import sys
5
+ import types
6
+ from pathlib import Path
7
+
8
+
9
+ def _stub_transformers() -> None:
10
+ if 'transformers' in sys.modules:
11
+ return
12
+ stub = types.ModuleType('transformers')
13
+ stub.AutoModelForCausalLM = type('AutoModelForCausalLM', (), {})
14
+ stub.AutoTokenizer = type('AutoTokenizer', (), {})
15
+ sys.modules['transformers'] = stub
16
+
17
+
18
+ def test_run_all_forwards_activation_collection_tuning(monkeypatch, tmp_path: Path) -> None:
19
+ from experiments import run_all
20
+
21
+ calls: list[list[str]] = []
22
+
23
+ def fake_run(command, **_kwargs):
24
+ calls.append(list(command))
25
+ return types.SimpleNamespace(returncode=0)
26
+
27
+ monkeypatch.setattr(run_all.subprocess, 'run', fake_run)
28
+ missing = tmp_path / 'not-created'
29
+ run_all.run(
30
+ 'experiments.collect_activations',
31
+ outputs=[missing],
32
+ resume=True,
33
+ extra_args=['--batch-size', '8', '--max-length', '192'],
34
+ )
35
+ assert calls
36
+ assert calls[0][-4:] == ['--batch-size', '8', '--max-length', '192']
37
+
38
+
39
+ def test_causal_checkpoint_helpers_roundtrip(tmp_path: Path) -> None:
40
+ _stub_transformers()
41
+ from experiments import run_causal
42
+
43
+ output = tmp_path / 'causal_results.csv'
44
+ rows = [
45
+ {'task_id': 'task-1', 'concept': 'mathematics', 'condition': 'sae_feature'},
46
+ {'task_id': 'task-1', 'concept': 'mathematics', 'condition': 'random_norm_matched'},
47
+ ]
48
+ run_causal._write_rows_atomic(output, rows)
49
+ loaded = run_causal._load_checkpoint_rows(output)
50
+ assert loaded == rows
51
+ assert run_causal._completion_marker(output).name == 'causal_results.csv.complete'
52
+ assert not output.with_suffix('.csv.tmp').exists()
53
+
54
+
55
+ def test_feature_set_checkpoint_helpers_roundtrip(tmp_path: Path) -> None:
56
+ _stub_transformers()
57
+ from experiments import run_feature_sets
58
+
59
+ output = tmp_path / 'feature_set_results.csv'
60
+ rows = [
61
+ {'task_id': 'task-1', 'set_size': '1', 'condition': 'sae_feature_set'},
62
+ {'task_id': 'task-1', 'set_size': '1', 'condition': 'random_norm_matched'},
63
+ ]
64
+ run_feature_sets._write_rows_atomic(output, rows)
65
+ with output.open(newline='', encoding='utf-8') as handle:
66
+ saved = list(csv.DictReader(handle))
67
+ assert saved == rows
68
+ assert run_feature_sets._load_checkpoint_rows(output) == rows
69
+ assert run_feature_sets._completion_marker(output).name == 'feature_set_results.csv.complete'
tests/test_ui_helpers.py CHANGED
@@ -49,7 +49,7 @@ def test_use_candidate_feature_returns_explicit_handoff_status() -> None:
49
  outputs = app.use_candidate_feature('21885')
50
  assert outputs[:4] == ('21885', '21885', '21885', '21885')
51
  assert 'Feature 21885 loaded' in outputs[4]
52
- assert 'Single-feature causal test' in outputs[4]
53
 
54
 
55
  def test_select_candidate_row_uses_feature_id_column() -> None:
@@ -106,7 +106,8 @@ def test_result_tables_hide_native_labels_in_favor_of_explicit_headings() -> Non
106
  assert app.discovery_table.show_label is False
107
  assert app.dose_table.show_label is False
108
  assert '.table-heading' in app.CSS
109
- assert 'margin: 0 0 -34px' in app.CSS
 
110
 
111
 
112
  def test_candidate_screen_markdown_is_explicitly_triage_only() -> None:
@@ -122,7 +123,6 @@ def test_candidate_screen_markdown_is_explicitly_triage_only() -> None:
122
  text = app._candidate_screen_metrics_markdown(result)
123
  assert 'triage screen' in text
124
  assert 'no random-control ensemble is spent here' in text
125
- assert 'Single-feature causal test' in text
126
 
127
 
128
  def test_candidate_screen_has_independent_target_and_multiselect() -> None:
 
49
  outputs = app.use_candidate_feature('21885')
50
  assert outputs[:4] == ('21885', '21885', '21885', '21885')
51
  assert 'Feature 21885 loaded' in outputs[4]
52
+ assert 'feature-level experiments' in outputs[4]
53
 
54
 
55
  def test_select_candidate_row_uses_feature_id_column() -> None:
 
106
  assert app.discovery_table.show_label is False
107
  assert app.dose_table.show_label is False
108
  assert '.table-heading' in app.CSS
109
+ assert 'margin: 2px 0 -32px' in app.CSS
110
+ assert 'font-size: 1.16rem' in app.CSS
111
 
112
 
113
  def test_candidate_screen_markdown_is_explicitly_triage_only() -> None:
 
123
  text = app._candidate_screen_metrics_markdown(result)
124
  assert 'triage screen' in text
125
  assert 'no random-control ensemble is spent here' in text
 
126
 
127
 
128
  def test_candidate_screen_has_independent_target_and_multiselect() -> None: