Initial FeatureLens v0.1.0

.github/workflows/ci.yml

@@ -0,0 +1,26 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install lightweight test dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install "torch>=2.8,<2.12" "huggingface_hub>=0.34,<2" "numpy>=2,<3" "pytest>=8.3,<10" "ruff>=0.9,<1"
+      - name: Ruff
+        run: python -m ruff check featurelens tests scripts experiments/split.py experiments/build_dataset.py
+      - name: Unit tests
+        run: python -m pytest -q
+      - name: Compile
+        run: python -m compileall -q app.py featurelens experiments scripts
+      - name: Release check
+        run: python scripts/release_check.py

.gitignore

@@ -0,0 +1,12 @@
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.ruff_cache/
+.venv/
+.env
+.DS_Store
+artifacts/activations/
+artifacts/*.npy
+artifacts/*.npz
+*.pt
+*.safetensors

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Archit Sharma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,13 +1,193 @@
 ---
 title: FeatureLens
-emoji: 📉
-colorFrom: yellow
-colorTo: blue
+emoji: 🔬
+colorFrom: blue
+colorTo: indigo
 sdk: gradio
-sdk_version: 6.24.0
-python_version: '3.12'
+python_version: "3.12"
 app_file: app.py
 pinned: false
+license: mit
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# FeatureLens — Causal Interpretability Workbench
+
+**FeatureLens asks one concrete question:**
+
+> Do sparse features that predict a concept also causally influence model behaviour?
+
+It uses **Qwen3-1.7B-Base** with the official **Qwen-Scope residual-stream sparse autoencoders (SAEs)**. The live Hugging Face Space inspects feature activations and performs causal residual-stream interventions; the offline pipeline measures whether predictive features remain predictive on held-out paraphrases and whether manipulating them actually changes downstream behavior.
+
+This is deliberately **not** a clone of an SAE viewer and does not depend on thesis code or thesis results.
+
+## What is different from a feature viewer?
+
+FeatureLens separates three claims that are often blurred together:
+
+1. **Representation:** does an SAE reconstruct the residual stream reasonably well?
+2. **Prediction:** does a feature distinguish a controlled concept on held-out prompts?
+3. **Causation:** does changing that feature coefficient alter downstream token probabilities or generation more than a norm-matched random perturbation?
+
+A feature can score highly on (2) and weakly on (3). FeatureLens treats that as a valid result rather than a failure.
+
+## Live workbench
+
+The Gradio app supports:
+
+- prompt input and token-level inspection;
+- early / middle / late residual layers: **4, 14, 26**;
+- strongest TopK SAE features at a selected token;
+- reconstruction cosine and normalized MSE;
+- feature **ablation**, **scaling**, or **injection**;
+- baseline vs modified greedy generation;
+- first-token probability table and Jensen-Shannon divergence;
+- optional target-continuation probability / log-probability delta;
+- optional concept hints loaded from real offline benchmark artifacts.
+
+### Reconstruction-preserving causal edit
+
+Qwen-Scope provides encoder and decoder weights. Let the original residual be `h`, the selected sparse feature activation be `z_i`, and the feature decoder direction be `d_i`.
+
+FeatureLens patches the **original** residual:
+
+```text
+ablate:   h' = h - z_i d_i
+scale α:  h' = h + (α - 1) z_i d_i
+inject δ: h' = h + δ d_i
+```
+
+It does **not** replace `h` with the SAE reconstruction. This keeps SAE reconstruction error out of the causal perturbation.
+
+## Offline experiment
+
+The repository ships a controlled benchmark with:
+
+- **224 discovery prompts** across 7 concepts;
+- **112 paraphrase pairs** kept together during splitting;
+- concepts: code, mathematics, positive sentiment, negative sentiment, French, factual entities, uncertainty;
+- **28 separate causal completion tasks**;
+- one forward pass captures all configured residual layers for each batch;
+- TopK sparse feature activations and dense residuals are saved separately.
+
+The evaluation computes:
+
+- SAE reconstruction cosine / NMSE;
+- actual active-feature count / sparsity;
+- held-out feature/concept AUROC and F1;
+- paraphrase TopK Jaccard and sparse-activation cosine;
+- layer-wise multinomial linear probes on dense residual states;
+- selected-feature ablation and 2× amplification;
+- target first-token probability, log-probability, rank and JS divergence;
+- **norm-matched random residual-direction controls**.
+
+Feature selection uses the **training split**. Held-out AUROC/F1 are reported afterward. Paraphrases from the same pair never cross the train/test boundary.
+
+## Run the full benchmark
+
+A CUDA machine is strongly recommended. The default pipeline downloads Qwen3-1.7B-Base plus the three selected Qwen-Scope layer checkpoints.
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+python experiments/run_all.py
+```
+
+Outputs are materialized under `artifacts/`:
+
+```text
+artifacts/
+├── activations/
+│   ├── metadata.json
+│   ├── residuals_layer4.npy
+│   ├── features_layer4.npz
+│   └── ...
+├── feature_catalog.csv
+├── layer_metrics.csv
+├── stability.csv
+├── causal_results.csv
+├── summary.json
+├── report.md
+└── figures/
+```
+
+`report.md` and `summary.json` are generated from the measured results. The code contains no fabricated benchmark numbers.
+
+## Hugging Face ZeroGPU deployment
+
+FeatureLens is intentionally a **Gradio SDK Space**, not a Docker Space, because ZeroGPU currently supports Gradio SDK Spaces.
+
+1. Create a new **Gradio** Space.
+2. In the Space hardware settings, select **ZeroGPU**.
+3. Push this repository as-is.
+4. No API key is required; both Qwen repositories are public.
+5. On first cold start, the Space downloads Qwen3-1.7B-Base and only SAE layers **4, 14 and 26**, not all 28 SAE files.
+
+The app uses the `spaces.GPU` decorator when available and falls back to ordinary execution locally. On a Hugging Face Space it eagerly places the model and selected SAEs on the ZeroGPU CUDA-emulation device at startup, matching Hugging Face's recommended loading pattern.
+
+Useful environment overrides:
+
+```text
+FEATURELENS_MODEL_ID=Qwen/Qwen3-1.7B-Base
+FEATURELENS_SAE_REPO=Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50
+FEATURELENS_LAYERS=4,14,26
+FEATURELENS_EAGER_LOAD=1
+FEATURELENS_SAE_DTYPE=float16
+FEATURELENS_MAX_NEW_TOKENS=32
+```
+
+For offline benchmarking, use `FEATURELENS_SAE_DTYPE=float32` (the scripts already use FP32 SAE weights by default).
+
+## Repository layout
+
+```text
+FeatureLens/
+├── app.py                         # Gradio / ZeroGPU workbench
+├── featurelens/
+│   ├── config.py                  # model + layer configuration
+│   ├── sae.py                     # TopK SAE loading / encode / decode
+│   ├── interventions.py           # causal deltas + random controls
+│   ├── runtime.py                 # hooks, generation, probability deltas
+│   ├── metrics.py                 # reconstruction + divergence metrics
+│   └── catalog.py                 # offline result integration
+├── experiments/
+│   ├── build_dataset.py
+│   ├── collect_activations.py
+│   ├── evaluate_features.py
+│   ├── run_causal.py
+│   ├── make_report.py
+│   └── run_all.py
+├── data/
+│   ├── prompts.jsonl
+│   └── causal_tasks.jsonl
+├── tests/
+├── scripts/release_check.py
+├── docs/
+└── research_config.json
+```
+
+## Methodological limitations
+
+- Qwen-Scope features are TopK sparse directions, not guaranteed monosemantic concepts.
+- The controlled benchmark is intentionally small enough to reproduce on modest research compute; it is not a universal feature ontology.
+- The live demo applies one residual edit at one selected prompt token. This is cleaner for causal attribution than repeatedly steering every generated token, but it may produce smaller behavioral effects.
+- A target string may tokenize into multiple tokens. The live workbench explicitly labels its target metric as the **first-token** probability in that case.
+- A causal effect can depend strongly on prompt, layer, feature scale, and downstream task. The report therefore includes raw per-task rows rather than only aggregate means.
+
+## Reproducibility and checks
+
+```bash
+python -m ruff check app.py featurelens experiments tests scripts
+python -m pytest -q
+python -m compileall -q app.py featurelens experiments scripts
+python scripts/release_check.py
+```
+
+## Resume-ready description
+
+> **FeatureLens — Causal Interpretability Workbench** | PyTorch, Qwen3, Sparse Autoencoders, Mechanistic Interpretability, Gradio  
+> Built an SAE-based interpretability system for Qwen3-1.7B that discovers concept-associated residual features, benchmarks held-out predictiveness and paraphrase stability against linear probes, and causally ablates/amplifies features with norm-matched controls while measuring downstream token-probability and generation shifts.
+
+## Acknowledgements
+
+FeatureLens builds on the open Qwen3 model and Qwen-Scope SAE checkpoints from the Qwen team. See the upstream model cards and Qwen-Scope technical report for model and SAE details and licensing requirements.

app.py

@@ -0,0 +1,288 @@
+from __future__ import annotations
+
+import gradio as gr
+
+from featurelens.config import SETTINGS
+from featurelens.hf_runtime import gpu
+from featurelens.runtime import RUNTIME
+
+CSS = """
+:root { --fl-blue:#2563eb; --fl-ink:#0f172a; --fl-muted:#64748b; }
+.gradio-container { max-width: 1280px !important; }
+.hero { padding: 12px 2px 6px; }
+.hero h1 { margin-bottom: 2px; font-size: 2.1rem; letter-spacing:-0.03em; }
+.hero p { color: var(--fl-muted); margin-top: 0; }
+.panel { border:1px solid #e2e8f0; border-radius:14px; padding:8px; }
+.token-wrap { display:flex; flex-wrap:wrap; gap:5px; padding:8px 2px; line-height:1.7; }
+.token { background:#f1f5f9; border:1px solid #e2e8f0; border-radius:6px; padding:2px 7px; font-family:ui-monospace,SFMono-Regular,monospace; font-size:12px; }
+.token.selected { background:#dbeafe; border:2px solid #2563eb; color:#1e3a8a; }
+.token sup { opacity:.55; margin-right:4px; }
+.small-note { color:#64748b; font-size:12px; }
+"""
+
+
+def _analysis_metrics_markdown(result) -> str:
+    return (
+        f"**Layer {result.layer} · token {result.token_index}**  \n"
+        f"Active SAE features: **{int(result.metrics['active_features'])}/{SETTINGS.sae_top_k}**  \n"
+        f"Reconstruction cosine: **{result.metrics['cosine']:.4f}**  \n"
+        f"Normalized MSE: **{result.metrics['nmse']:.4f}**"
+    )
+
+
+def _intervention_metrics_markdown(result) -> str:
+    target = 'No target continuation supplied.'
+    if result.baseline_target_prob is not None:
+        warning = ''
+        if result.target_token_count > 1:
+            warning = (
+                f"  \n_Note: target text tokenizes to {result.target_token_count} tokens; "
+                'the displayed probability is for its first token only._'
+            )
+        target = (
+            f"Target first token: `{result.target_token}`  \n"
+            f"Baseline p: **{result.baseline_target_prob:.6f}** · "
+            f"Modified p: **{result.modified_target_prob:.6f}**  \n"
+            f"Δ log p: **{result.target_logprob_delta:+.4f}**{warning}"
+        )
+    return (
+        f"Original feature activation: **{result.feature_activation:.4f}**  \n"
+        f"Δ feature coefficient: **{result.delta_activation:+.4f}**  \n"
+        f"Residual perturbation L2: **{result.perturbation_norm:.4f}**  \n"
+        f"Next-token JS divergence: **{result.js_divergence:.6f}**  \n\n"
+        f"{target}"
+    )
+
+
+@gpu(duration=45)
+def analyze_prompt(prompt: str, layer: int, token_index: int, top_n: int):
+    if not prompt.strip():
+        raise gr.Error('Enter a prompt first.')
+    result = RUNTIME.analyze(prompt, int(layer), int(token_index), int(top_n))
+    choices = [str(int(row[1])) for row in result.rows]
+    feature_update = gr.update(choices=choices, value=choices[0] if choices else None)
+    return (
+        RUNTIME.token_html(result.tokens, result.token_index),
+        result.rows,
+        feature_update,
+        _analysis_metrics_markdown(result),
+    )
+
+
+@gpu(duration=60)
+def run_intervention(
+    prompt: str,
+    layer: int,
+    token_index: int,
+    feature_id: str,
+    mode: str,
+    coefficient: float,
+    target_text: str,
+    max_new_tokens: int,
+):
+    if not prompt.strip():
+        raise gr.Error('Enter a prompt first.')
+    if feature_id is None or str(feature_id).strip() == '':
+        raise gr.Error('Choose or enter a feature id.')
+    try:
+        fid = int(float(feature_id))
+    except ValueError as exc:
+        raise gr.Error('Feature id must be an integer.') from exc
+    result = RUNTIME.intervene(
+        text=prompt,
+        layer=int(layer),
+        token_index=int(token_index),
+        feature_id=fid,
+        mode=mode,
+        coefficient=float(coefficient),
+        target_text=target_text,
+        max_new_tokens=int(max_new_tokens),
+    )
+    return (
+        result.baseline_text,
+        result.modified_text,
+        _intervention_metrics_markdown(result),
+        result.top_token_rows,
+    )
+
+
+def mode_help(mode: str):
+    if mode == 'ablate':
+        return gr.update(
+            value=0.0,
+            interactive=False,
+            label='Coefficient (unused for ablation)',
+            info='Ablation sets the selected TopK feature coefficient to zero.',
+        )
+    if mode == 'scale':
+        return gr.update(
+            value=2.0,
+            interactive=True,
+            label='Feature multiplier',
+            info='1.0 = unchanged, 0 = ablate, 2.0 = double the original coefficient.',
+        )
+    return gr.update(
+        value=5.0,
+        interactive=True,
+        label='Additive feature coefficient',
+        info='Adds this amount along the feature decoder direction, even if the feature is inactive.',
+    )
+
+
+with gr.Blocks(title='FeatureLens — Causal Interpretability Workbench') as demo:
+    gr.HTML(
+        '<div class="hero"><h1>FeatureLens</h1>'
+        '<p>Causal sparse-feature interpretability for Qwen3-1.7B: inspect → intervene → measure.</p></div>'
+    )
+    gr.Markdown(
+        f"Model: `{SETTINGS.model_id}` · Qwen-Scope TopK SAE · layers "
+        f"`{', '.join(map(str, SETTINGS.layers))}` · width `{SETTINGS.sae_width:,}`"
+    )
+
+    with gr.Tab('Workbench'):
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=2):
+                prompt = gr.Textbox(
+                    label='Prompt',
+                    lines=7,
+                    value='The derivative of x squared is',
+                    placeholder='Enter a prompt to inspect...',
+                )
+                with gr.Row():
+                    layer = gr.Dropdown(
+                        choices=list(SETTINGS.layers),
+                        value=SETTINGS.layers[1] if len(SETTINGS.layers) > 1 else SETTINGS.layers[0],
+                        label='Residual layer',
+                    )
+                    token_index = gr.Number(
+                        value=-1,
+                        precision=0,
+                        label='Token index',
+                        info='Use -1 for the final prompt token. Analyze once to see all token indices.',
+                    )
+                    top_n = gr.Slider(5, 20, value=12, step=1, label='Top features')
+                analyze_btn = gr.Button('Inspect SAE features', variant='primary')
+                token_view = gr.HTML(
+                    '<div class="small-note">Token positions will appear here after analysis.</div>'
+                )
+                analysis_metrics = gr.Markdown()
+
+            with gr.Column(scale=3):
+                feature_table = gr.Dataframe(
+                    headers=['Rank', 'Feature id', 'Activation', 'Offline concept hint'],
+                    datatype=['number', 'number', 'number', 'str'],
+                    interactive=False,
+                    label='Strongest TopK features at the selected token',
+                    wrap=True,
+                )
+
+        gr.Markdown('### Causal intervention')
+        gr.Markdown(
+            'The edit is **reconstruction-preserving**: FeatureLens changes only the selected SAE '
+            'coefficient and adds the decoded delta to the *original* residual stream. It does not '
+            'replace the residual with an SAE reconstruction.'
+        )
+        with gr.Row(equal_height=False):
+            with gr.Column(scale=2):
+                feature_id = gr.Dropdown(
+                    choices=[],
+                    allow_custom_value=True,
+                    label='Feature id',
+                    info='Analyze first to populate the strongest features, or enter any id 0–32767.',
+                )
+                mode = gr.Radio(
+                    choices=['ablate', 'scale', 'inject'],
+                    value='ablate',
+                    label='Intervention',
+                )
+                coefficient = gr.Number(
+                    value=0.0,
+                    interactive=False,
+                    label='Coefficient (unused for ablation)',
+                )
+                target_text = gr.Textbox(
+                    label='Optional target continuation',
+                    placeholder='e.g. 2x',
+                    info='FeatureLens reports the first-token probability shift for this continuation.',
+                )
+                max_new = gr.Slider(
+                    4,
+                    SETTINGS.max_new_tokens,
+                    value=min(20, SETTINGS.max_new_tokens),
+                    step=1,
+                    label='Max new tokens',
+                )
+                intervene_btn = gr.Button('Run baseline vs modified', variant='primary')
+                intervention_metrics = gr.Markdown()
+
+            with gr.Column(scale=3):
+                with gr.Row():
+                    baseline_out = gr.Textbox(label='Baseline output', lines=8, interactive=False)
+                    modified_out = gr.Textbox(label='Modified output', lines=8, interactive=False)
+                token_prob_table = gr.Dataframe(
+                    headers=['Token', 'Baseline p', 'Modified p', 'Δ probability'],
+                    datatype=['str', 'number', 'number', 'number'],
+                    interactive=False,
+                    label='First generated token distribution',
+                )
+
+        analyze_btn.click(
+            analyze_prompt,
+            inputs=[prompt, layer, token_index, top_n],
+            outputs=[token_view, feature_table, feature_id, analysis_metrics],
+        )
+        mode.change(mode_help, inputs=[mode], outputs=[coefficient])
+        intervene_btn.click(
+            run_intervention,
+            inputs=[
+                prompt,
+                layer,
+                token_index,
+                feature_id,
+                mode,
+                coefficient,
+                target_text,
+                max_new,
+            ],
+            outputs=[baseline_out, modified_out, intervention_metrics, token_prob_table],
+        )
+
+    with gr.Tab('Offline benchmark'):
+        benchmark_md = gr.Markdown(RUNTIME.catalog.benchmark_markdown())
+        gr.Markdown(
+            'The offline pipeline evaluates feature/concept AUROC + F1, reconstruction quality, '
+            'paraphrase stability, layer-wise residual linear probes, causal ablation/amplification, '
+            'and norm-matched random-direction controls. Results are loaded from `artifacts/` and '
+            'are never hard-coded into the demo.'
+        )
+
+    with gr.Tab('Method'):
+        gr.Markdown(
+            r"""
+### What the intervention means
+
+For residual vector $h$, sparse code $z$, decoder column $d_i$, and selected feature $i$:
+
+- **Ablate:** $h' = h - z_i d_i$
+- **Scale by $\alpha$:** $h' = h + (\alpha - 1) z_i d_i$
+- **Inject $\delta$:** $h' = h + \delta d_i$
+
+This is equivalent to editing the SAE reconstruction by the chosen feature delta while retaining
+$h$ itself, so SAE reconstruction error is not injected as a confound. The modified residual is
+patched at one selected **prompt token**; downstream generation is then allowed to evolve normally.
+
+### What FeatureLens does *not* claim
+
+A high feature/concept AUROC is correlational evidence. A causal claim requires downstream effects
+under intervention, held-out prompts, and comparison with controls. The offline report is designed
+to make a weak or null causal result visible rather than hide it.
+"""
+        )
+
+    gr.Markdown(
+        'Built with PyTorch, Transformers, Qwen3-1.7B-Base and Qwen-Scope SAEs. '
+        'This project is independent of any thesis dataset or thesis code.'
+    )
+
+if __name__ == '__main__':
+    demo.queue(default_concurrency_limit=1).launch(css=CSS)

artifacts/README.md

@@ -0,0 +1,13 @@
+# Generated artifacts
+
+This directory intentionally ships without invented results.
+
+Run:
+
+```bash
+python experiments/run_all.py
+```
+
+to create activation caches, `feature_catalog.csv`, `layer_metrics.csv`, `stability.csv`, `causal_results.csv`, `summary.json`, `report.md`, and figures.
+
+Large activation arrays are ignored by Git. Commit the small CSV/JSON/report/figure outputs if you want the live Space to show benchmark-derived feature hints and the benchmark summary.

data/causal_tasks.jsonl

@@ -0,0 +1,28 @@
+{"id": 0, "concept": "positive_sentiment", "prompt": "Sentiment: The meal was wonderful and the staff were kind. Label:", "target": " positive"}
+{"id": 1, "concept": "positive_sentiment", "prompt": "Sentiment: I loved the thoughtful, beautifully written novel. Label:", "target": " positive"}
+{"id": 2, "concept": "positive_sentiment", "prompt": "Sentiment: The service was fast and extremely helpful. Label:", "target": " positive"}
+{"id": 3, "concept": "positive_sentiment", "prompt": "Sentiment: The journey was comfortable and enjoyable. Label:", "target": " positive"}
+{"id": 4, "concept": "negative_sentiment", "prompt": "Sentiment: The meal was awful and the staff were rude. Label:", "target": " negative"}
+{"id": 5, "concept": "negative_sentiment", "prompt": "Sentiment: I hated the tedious, badly written novel. Label:", "target": " negative"}
+{"id": 6, "concept": "negative_sentiment", "prompt": "Sentiment: The service was slow and completely unhelpful. Label:", "target": " negative"}
+{"id": 7, "concept": "negative_sentiment", "prompt": "Sentiment: The journey was stressful and unpleasant. Label:", "target": " negative"}
+{"id": 8, "concept": "mathematics", "prompt": "2 + 3 =", "target": " 5"}
+{"id": 9, "concept": "mathematics", "prompt": "7 - 4 =", "target": " 3"}
+{"id": 10, "concept": "mathematics", "prompt": "6 * 2 =", "target": " 12"}
+{"id": 11, "concept": "mathematics", "prompt": "The square root of 81 is", "target": " 9"}
+{"id": 12, "concept": "code", "prompt": "Python function declaration keyword:", "target": " def"}
+{"id": 13, "concept": "code", "prompt": "In Python, an exception handler begins with the keyword", "target": " except"}
+{"id": 14, "concept": "code", "prompt": "SQL keyword used to retrieve rows:", "target": " SELECT"}
+{"id": 15, "concept": "code", "prompt": "A Python conditional branch commonly starts with", "target": " if"}
+{"id": 16, "concept": "french_language", "prompt": "Translate 'hello' into French:", "target": " bonjour"}
+{"id": 17, "concept": "french_language", "prompt": "Translate 'thank you' into French:", "target": " merci"}
+{"id": 18, "concept": "french_language", "prompt": "Translate 'yes' into French:", "target": " oui"}
+{"id": 19, "concept": "french_language", "prompt": "Translate 'good evening' into French:", "target": " bonsoir"}
+{"id": 20, "concept": "factual_entities", "prompt": "The scientist associated with radium research, Marie", "target": " Curie"}
+{"id": 21, "concept": "factual_entities", "prompt": "The composer of the Fifth Symphony, Ludwig van", "target": " Beethoven"}
+{"id": 22, "concept": "factual_entities", "prompt": "The computer scientist known for the Turing machine, Alan", "target": " Turing"}
+{"id": 23, "concept": "factual_entities", "prompt": "The Japanese city famous for many historic temples,", "target": " Kyoto"}
+{"id": 24, "concept": "uncertainty", "prompt": "The evidence is insufficient. The answer is", "target": " unknown"}
+{"id": 25, "concept": "uncertainty", "prompt": "The passage never states the value, so it is", "target": " unknown"}
+{"id": 26, "concept": "uncertainty", "prompt": "There is not enough information to determine the result. It remains", "target": " uncertain"}
+{"id": 27, "concept": "uncertainty", "prompt": "No reliable conclusion can be drawn; the outcome is", "target": " unclear"}

data/prompts.jsonl

@@ -0,0 +1,224 @@
+{"id": 0, "concept": "code", "pair_id": "code-00", "variant": 0, "text": "Write a Python function that returns the larger of two integers."}
+{"id": 1, "concept": "code", "pair_id": "code-00", "variant": 1, "text": "In Python, create a function that chooses the maximum of two integer inputs."}
+{"id": 2, "concept": "code", "pair_id": "code-01", "variant": 0, "text": "Explain what a recursive function does in a program."}
+{"id": 3, "concept": "code", "pair_id": "code-01", "variant": 1, "text": "Describe recursion in the context of a programming function."}
+{"id": 4, "concept": "code", "pair_id": "code-02", "variant": 0, "text": "Show a JavaScript loop that prints numbers from one to five."}
+{"id": 5, "concept": "code", "pair_id": "code-02", "variant": 1, "text": "Give JavaScript code for iterating over the integers 1 through 5 and logging them."}
+{"id": 6, "concept": "code", "pair_id": "code-03", "variant": 0, "text": "What is the purpose of a hash map in software?"}
+{"id": 7, "concept": "code", "pair_id": "code-03", "variant": 1, "text": "Explain why programmers use hash maps or dictionaries."}
+{"id": 8, "concept": "code", "pair_id": "code-04", "variant": 0, "text": "Write SQL that selects every row from a table named users."}
+{"id": 9, "concept": "code", "pair_id": "code-04", "variant": 1, "text": "Provide a SQL query to retrieve all records in the users table."}
+{"id": 10, "concept": "code", "pair_id": "code-05", "variant": 0, "text": "How do I catch an exception in Python?"}
+{"id": 11, "concept": "code", "pair_id": "code-05", "variant": 1, "text": "Show the Python syntax for handling an exception with try and except."}
+{"id": 12, "concept": "code", "pair_id": "code-06", "variant": 0, "text": "Explain the difference between a list and a tuple in Python."}
+{"id": 13, "concept": "code", "pair_id": "code-06", "variant": 1, "text": "Compare Python lists with tuples."}
+{"id": 14, "concept": "code", "pair_id": "code-07", "variant": 0, "text": "Write a function that checks whether a string is a palindrome."}
+{"id": 15, "concept": "code", "pair_id": "code-07", "variant": 1, "text": "Create code that tests if text reads the same forwards and backwards."}
+{"id": 16, "concept": "code", "pair_id": "code-08", "variant": 0, "text": "What does git rebase do?"}
+{"id": 17, "concept": "code", "pair_id": "code-08", "variant": 1, "text": "Describe the effect of rebasing a Git branch."}
+{"id": 18, "concept": "code", "pair_id": "code-09", "variant": 0, "text": "How can a REST API return JSON?"}
+{"id": 19, "concept": "code", "pair_id": "code-09", "variant": 1, "text": "Explain how a web API endpoint sends a JSON response."}
+{"id": 20, "concept": "code", "pair_id": "code-10", "variant": 0, "text": "Write pseudocode for binary search."}
+{"id": 21, "concept": "code", "pair_id": "code-10", "variant": 1, "text": "Describe the binary-search algorithm as pseudocode."}
+{"id": 22, "concept": "code", "pair_id": "code-11", "variant": 0, "text": "What is a class constructor used for?"}
+{"id": 23, "concept": "code", "pair_id": "code-11", "variant": 1, "text": "Explain the role of a constructor when creating an object."}
+{"id": 24, "concept": "code", "pair_id": "code-12", "variant": 0, "text": "Give a regular expression that matches a basic email-like string."}
+{"id": 25, "concept": "code", "pair_id": "code-12", "variant": 1, "text": "Show a regex pattern for a simple email address format."}
+{"id": 26, "concept": "code", "pair_id": "code-13", "variant": 0, "text": "Why is time complexity important when comparing algorithms?"}
+{"id": 27, "concept": "code", "pair_id": "code-13", "variant": 1, "text": "Explain why Big-O runtime matters in algorithm analysis."}
+{"id": 28, "concept": "code", "pair_id": "code-14", "variant": 0, "text": "How do unit tests help a codebase?"}
+{"id": 29, "concept": "code", "pair_id": "code-14", "variant": 1, "text": "Describe the purpose of automated unit testing in software."}
+{"id": 30, "concept": "code", "pair_id": "code-15", "variant": 0, "text": "Write Python that sorts a list of dictionaries by an age key."}
+{"id": 31, "concept": "code", "pair_id": "code-15", "variant": 1, "text": "Show how to order Python dictionaries in a list according to their age field."}
+{"id": 32, "concept": "mathematics", "pair_id": "mathematics-00", "variant": 0, "text": "Find the derivative of x cubed plus two x."}
+{"id": 33, "concept": "mathematics", "pair_id": "mathematics-00", "variant": 1, "text": "Differentiate the function x^3 + 2x."}
+{"id": 34, "concept": "mathematics", "pair_id": "mathematics-01", "variant": 0, "text": "Solve the equation 3x + 5 = 20."}
+{"id": 35, "concept": "mathematics", "pair_id": "mathematics-01", "variant": 1, "text": "Find x when three x plus five equals twenty."}
+{"id": 36, "concept": "mathematics", "pair_id": "mathematics-02", "variant": 0, "text": "What is the integral of cosine x?"}
+{"id": 37, "concept": "mathematics", "pair_id": "mathematics-02", "variant": 1, "text": "Compute an antiderivative of cos(x)."}
+{"id": 38, "concept": "mathematics", "pair_id": "mathematics-03", "variant": 0, "text": "Factor the polynomial x squared minus nine."}
+{"id": 39, "concept": "mathematics", "pair_id": "mathematics-03", "variant": 1, "text": "Rewrite x^2 - 9 as a product of factors."}
+{"id": 40, "concept": "mathematics", "pair_id": "mathematics-04", "variant": 0, "text": "Calculate the mean of 4, 7, 9, and 10."}
+{"id": 41, "concept": "mathematics", "pair_id": "mathematics-04", "variant": 1, "text": "What is the arithmetic average of 4, 7, 9, 10?"}
+{"id": 42, "concept": "mathematics", "pair_id": "mathematics-05", "variant": 0, "text": "Explain the Pythagorean theorem."}
+{"id": 43, "concept": "mathematics", "pair_id": "mathematics-05", "variant": 1, "text": "State the relationship among the sides of a right triangle."}
+{"id": 44, "concept": "mathematics", "pair_id": "mathematics-06", "variant": 0, "text": "What is the determinant of a two by two matrix?"}
+{"id": 45, "concept": "mathematics", "pair_id": "mathematics-06", "variant": 1, "text": "Give the formula for the determinant of a 2x2 matrix."}
+{"id": 46, "concept": "mathematics", "pair_id": "mathematics-07", "variant": 0, "text": "Simplify two to the third power times two squared."}
+{"id": 47, "concept": "mathematics", "pair_id": "mathematics-07", "variant": 1, "text": "Reduce 2^3 multiplied by 2^2 using exponent rules."}
+{"id": 48, "concept": "mathematics", "pair_id": "mathematics-08", "variant": 0, "text": "Convert one half into a percentage."}
+{"id": 49, "concept": "mathematics", "pair_id": "mathematics-08", "variant": 1, "text": "Express 1/2 as a percent."}
+{"id": 50, "concept": "mathematics", "pair_id": "mathematics-09", "variant": 0, "text": "What is the probability of heads on a fair coin?"}
+{"id": 51, "concept": "mathematics", "pair_id": "mathematics-09", "variant": 1, "text": "For a fair coin, calculate the chance of flipping heads."}
+{"id": 52, "concept": "mathematics", "pair_id": "mathematics-10", "variant": 0, "text": "Solve x squared equals sixteen."}
+{"id": 53, "concept": "mathematics", "pair_id": "mathematics-10", "variant": 1, "text": "Find the real values of x satisfying x^2 = 16."}
+{"id": 54, "concept": "mathematics", "pair_id": "mathematics-11", "variant": 0, "text": "What is the slope between points (1,2) and (3,6)?"}
+{"id": 55, "concept": "mathematics", "pair_id": "mathematics-11", "variant": 1, "text": "Calculate the gradient of the line through (1,2) and (3,6)."}
+{"id": 56, "concept": "mathematics", "pair_id": "mathematics-12", "variant": 0, "text": "Explain what a prime number is."}
+{"id": 57, "concept": "mathematics", "pair_id": "mathematics-12", "variant": 1, "text": "Define a prime integer."}
+{"id": 58, "concept": "mathematics", "pair_id": "mathematics-13", "variant": 0, "text": "Compute the dot product of vectors (1,2) and (3,4)."}
+{"id": 59, "concept": "mathematics", "pair_id": "mathematics-13", "variant": 1, "text": "Find (1,2) · (3,4)."}
+{"id": 60, "concept": "mathematics", "pair_id": "mathematics-14", "variant": 0, "text": "What is log base ten of one thousand?"}
+{"id": 61, "concept": "mathematics", "pair_id": "mathematics-14", "variant": 1, "text": "Evaluate log_10(1000)."}
+{"id": 62, "concept": "mathematics", "pair_id": "mathematics-15", "variant": 0, "text": "A circle has radius three. What is its area?"}
+{"id": 63, "concept": "mathematics", "pair_id": "mathematics-15", "variant": 1, "text": "Calculate the area of a circle whose radius is 3."}
+{"id": 64, "concept": "positive_sentiment", "pair_id": "positive_sentiment-00", "variant": 0, "text": "The film was delightful, clever, and beautifully acted."}
+{"id": 65, "concept": "positive_sentiment", "pair_id": "positive_sentiment-00", "variant": 1, "text": "I found the movie charming, smart, and wonderfully performed."}
+{"id": 66, "concept": "positive_sentiment", "pair_id": "positive_sentiment-01", "variant": 0, "text": "This restaurant served an excellent meal and the staff were kind."}
+{"id": 67, "concept": "positive_sentiment", "pair_id": "positive_sentiment-01", "variant": 1, "text": "The food was fantastic and the service team was genuinely friendly."}
+{"id": 68, "concept": "positive_sentiment", "pair_id": "positive_sentiment-02", "variant": 0, "text": "I am very pleased with how reliable this laptop has been."}
+{"id": 69, "concept": "positive_sentiment", "pair_id": "positive_sentiment-02", "variant": 1, "text": "This laptop has worked dependably and I am extremely satisfied with it."}
+{"id": 70, "concept": "positive_sentiment", "pair_id": "positive_sentiment-03", "variant": 0, "text": "The concert was energetic and unforgettable."}
+{"id": 71, "concept": "positive_sentiment", "pair_id": "positive_sentiment-03", "variant": 1, "text": "I had an amazing time at the lively, memorable concert."}
+{"id": 72, "concept": "positive_sentiment", "pair_id": "positive_sentiment-04", "variant": 0, "text": "Her explanation was clear and genuinely helpful."}
+{"id": 73, "concept": "positive_sentiment", "pair_id": "positive_sentiment-04", "variant": 1, "text": "She explained the topic in a useful and easy-to-understand way."}
+{"id": 74, "concept": "positive_sentiment", "pair_id": "positive_sentiment-05", "variant": 0, "text": "The hotel room was spotless and comfortable."}
+{"id": 75, "concept": "positive_sentiment", "pair_id": "positive_sentiment-05", "variant": 1, "text": "Our room was exceptionally clean and pleasant to stay in."}
+{"id": 76, "concept": "positive_sentiment", "pair_id": "positive_sentiment-06", "variant": 0, "text": "I loved the book from beginning to end."}
+{"id": 77, "concept": "positive_sentiment", "pair_id": "positive_sentiment-06", "variant": 1, "text": "The novel kept me delighted all the way through."}
+{"id": 78, "concept": "positive_sentiment", "pair_id": "positive_sentiment-07", "variant": 0, "text": "The new update makes the app much easier to use."}
+{"id": 79, "concept": "positive_sentiment", "pair_id": "positive_sentiment-07", "variant": 1, "text": "After the update, the application feels significantly more convenient."}
+{"id": 80, "concept": "positive_sentiment", "pair_id": "positive_sentiment-08", "variant": 0, "text": "Their customer support solved my problem quickly."}
+{"id": 81, "concept": "positive_sentiment", "pair_id": "positive_sentiment-08", "variant": 1, "text": "Support handled the issue fast and left me very happy."}
+{"id": 82, "concept": "positive_sentiment", "pair_id": "positive_sentiment-09", "variant": 0, "text": "The hike had spectacular views and perfect weather."}
+{"id": 83, "concept": "positive_sentiment", "pair_id": "positive_sentiment-09", "variant": 1, "text": "We enjoyed gorgeous scenery and wonderful conditions on the hike."}
+{"id": 84, "concept": "positive_sentiment", "pair_id": "positive_sentiment-10", "variant": 0, "text": "The presentation was engaging and well organized."}
+{"id": 85, "concept": "positive_sentiment", "pair_id": "positive_sentiment-10", "variant": 1, "text": "I enjoyed the talk because it was compelling and structured clearly."}
+{"id": 86, "concept": "positive_sentiment", "pair_id": "positive_sentiment-11", "variant": 0, "text": "This camera takes sharp photos and feels great to use."}
+{"id": 87, "concept": "positive_sentiment", "pair_id": "positive_sentiment-11", "variant": 1, "text": "The camera produces crisp images and has a satisfying design."}
+{"id": 88, "concept": "positive_sentiment", "pair_id": "positive_sentiment-12", "variant": 0, "text": "The workshop exceeded my expectations."}
+{"id": 89, "concept": "positive_sentiment", "pair_id": "positive_sentiment-12", "variant": 1, "text": "I was impressed because the workshop was even better than I expected."}
+{"id": 90, "concept": "positive_sentiment", "pair_id": "positive_sentiment-13", "variant": 0, "text": "Dinner turned out wonderfully and everyone enjoyed it."}
+{"id": 91, "concept": "positive_sentiment", "pair_id": "positive_sentiment-13", "variant": 1, "text": "The evening meal was a success and all of us had a great time."}
+{"id": 92, "concept": "positive_sentiment", "pair_id": "positive_sentiment-14", "variant": 0, "text": "The museum exhibition was fascinating."}
+{"id": 93, "concept": "positive_sentiment", "pair_id": "positive_sentiment-14", "variant": 1, "text": "I thought the exhibition was deeply interesting and rewarding."}
+{"id": 94, "concept": "positive_sentiment", "pair_id": "positive_sentiment-15", "variant": 0, "text": "The train journey was smooth and relaxing."}
+{"id": 95, "concept": "positive_sentiment", "pair_id": "positive_sentiment-15", "variant": 1, "text": "The trip by train felt easy, calm, and comfortable."}
+{"id": 96, "concept": "negative_sentiment", "pair_id": "negative_sentiment-00", "variant": 0, "text": "The film was tedious, confusing, and badly acted."}
+{"id": 97, "concept": "negative_sentiment", "pair_id": "negative_sentiment-00", "variant": 1, "text": "I found the movie boring, incoherent, and poorly performed."}
+{"id": 98, "concept": "negative_sentiment", "pair_id": "negative_sentiment-01", "variant": 0, "text": "This restaurant served cold food and the staff were rude."}
+{"id": 99, "concept": "negative_sentiment", "pair_id": "negative_sentiment-01", "variant": 1, "text": "The meal arrived cold and the service team behaved unpleasantly."}
+{"id": 100, "concept": "negative_sentiment", "pair_id": "negative_sentiment-02", "variant": 0, "text": "I am disappointed by how unreliable this laptop has been."}
+{"id": 101, "concept": "negative_sentiment", "pair_id": "negative_sentiment-02", "variant": 1, "text": "This laptop keeps failing and I am extremely dissatisfied with it."}
+{"id": 102, "concept": "negative_sentiment", "pair_id": "negative_sentiment-03", "variant": 0, "text": "The concert was chaotic and forgettable."}
+{"id": 103, "concept": "negative_sentiment", "pair_id": "negative_sentiment-03", "variant": 1, "text": "I had a miserable time at the disorganized, dull concert."}
+{"id": 104, "concept": "negative_sentiment", "pair_id": "negative_sentiment-04", "variant": 0, "text": "Her explanation was unclear and unhelpful."}
+{"id": 105, "concept": "negative_sentiment", "pair_id": "negative_sentiment-04", "variant": 1, "text": "She explained the topic in a confusing way that did not help me."}
+{"id": 106, "concept": "negative_sentiment", "pair_id": "negative_sentiment-05", "variant": 0, "text": "The hotel room was dirty and uncomfortable."}
+{"id": 107, "concept": "negative_sentiment", "pair_id": "negative_sentiment-05", "variant": 1, "text": "Our room was unpleasant, unclean, and difficult to relax in."}
+{"id": 108, "concept": "negative_sentiment", "pair_id": "negative_sentiment-06", "variant": 0, "text": "I regretted reading the book."}
+{"id": 109, "concept": "negative_sentiment", "pair_id": "negative_sentiment-06", "variant": 1, "text": "The novel was a frustrating waste of my time."}
+{"id": 110, "concept": "negative_sentiment", "pair_id": "negative_sentiment-07", "variant": 0, "text": "The new update makes the app harder to use."}
+{"id": 111, "concept": "negative_sentiment", "pair_id": "negative_sentiment-07", "variant": 1, "text": "After the update, the application feels significantly more awkward."}
+{"id": 112, "concept": "negative_sentiment", "pair_id": "negative_sentiment-08", "variant": 0, "text": "Their customer support ignored my problem."}
+{"id": 113, "concept": "negative_sentiment", "pair_id": "negative_sentiment-08", "variant": 1, "text": "Support failed to resolve the issue and left me angry."}
+{"id": 114, "concept": "negative_sentiment", "pair_id": "negative_sentiment-09", "variant": 0, "text": "The hike had awful weather and disappointing views."}
+{"id": 115, "concept": "negative_sentiment", "pair_id": "negative_sentiment-09", "variant": 1, "text": "We dealt with terrible conditions and underwhelming scenery on the hike."}
+{"id": 116, "concept": "negative_sentiment", "pair_id": "negative_sentiment-10", "variant": 0, "text": "The presentation was dull and poorly organized."}
+{"id": 117, "concept": "negative_sentiment", "pair_id": "negative_sentiment-10", "variant": 1, "text": "I disliked the talk because it was tedious and structured badly."}
+{"id": 118, "concept": "negative_sentiment", "pair_id": "negative_sentiment-11", "variant": 0, "text": "This camera takes blurry photos and feels cheap."}
+{"id": 119, "concept": "negative_sentiment", "pair_id": "negative_sentiment-11", "variant": 1, "text": "The camera produces soft images and has a flimsy design."}
+{"id": 120, "concept": "negative_sentiment", "pair_id": "negative_sentiment-12", "variant": 0, "text": "The workshop fell far below my expectations."}
+{"id": 121, "concept": "negative_sentiment", "pair_id": "negative_sentiment-12", "variant": 1, "text": "I was disappointed because the workshop was much worse than I expected."}
+{"id": 122, "concept": "negative_sentiment", "pair_id": "negative_sentiment-13", "variant": 0, "text": "Dinner went badly and nobody enjoyed it."}
+{"id": 123, "concept": "negative_sentiment", "pair_id": "negative_sentiment-13", "variant": 1, "text": "The evening meal was a failure and all of us had a poor time."}
+{"id": 124, "concept": "negative_sentiment", "pair_id": "negative_sentiment-14", "variant": 0, "text": "The museum exhibition was painfully boring."}
+{"id": 125, "concept": "negative_sentiment", "pair_id": "negative_sentiment-14", "variant": 1, "text": "I thought the exhibition was dull and unrewarding."}
+{"id": 126, "concept": "negative_sentiment", "pair_id": "negative_sentiment-15", "variant": 0, "text": "The train journey was stressful and uncomfortable."}
+{"id": 127, "concept": "negative_sentiment", "pair_id": "negative_sentiment-15", "variant": 1, "text": "The trip by train felt frustrating, noisy, and unpleasant."}
+{"id": 128, "concept": "french_language", "pair_id": "french_language-00", "variant": 0, "text": "Bonjour, comment allez-vous aujourd’hui ?"}
+{"id": 129, "concept": "french_language", "pair_id": "french_language-00", "variant": 1, "text": "Salut, comment vas-tu aujourd’hui ?"}
+{"id": 130, "concept": "french_language", "pair_id": "french_language-01", "variant": 0, "text": "Je voudrais réserver une table pour deux personnes."}
+{"id": 131, "concept": "french_language", "pair_id": "french_language-01", "variant": 1, "text": "Puis-je réserver une table pour deux, s’il vous plaît ?"}
+{"id": 132, "concept": "french_language", "pair_id": "french_language-02", "variant": 0, "text": "La bibliothèque ferme à dix-huit heures."}
+{"id": 133, "concept": "french_language", "pair_id": "french_language-02", "variant": 1, "text": "La bibliothèque est ouverte jusqu’à dix-huit heures."}
+{"id": 134, "concept": "french_language", "pair_id": "french_language-03", "variant": 0, "text": "Ce livre raconte une histoire très intéressante."}
+{"id": 135, "concept": "french_language", "pair_id": "french_language-03", "variant": 1, "text": "L’histoire racontée dans ce livre est vraiment intéressante."}
+{"id": 136, "concept": "french_language", "pair_id": "french_language-04", "variant": 0, "text": "Nous allons prendre le train demain matin."}
+{"id": 137, "concept": "french_language", "pair_id": "french_language-04", "variant": 1, "text": "Demain matin, nous voyagerons en train."}
+{"id": 138, "concept": "french_language", "pair_id": "french_language-05", "variant": 0, "text": "Pouvez-vous m’indiquer le chemin vers la gare ?"}
+{"id": 139, "concept": "french_language", "pair_id": "french_language-05", "variant": 1, "text": "Comment puis-je aller jusqu’à la gare ?"}
+{"id": 140, "concept": "french_language", "pair_id": "french_language-06", "variant": 0, "text": "J’aime apprendre de nouvelles langues."}
+{"id": 141, "concept": "french_language", "pair_id": "french_language-06", "variant": 1, "text": "Apprendre des langues nouvelles me plaît beaucoup."}
+{"id": 142, "concept": "french_language", "pair_id": "french_language-07", "variant": 0, "text": "Le temps est magnifique au bord de la mer."}
+{"id": 143, "concept": "french_language", "pair_id": "french_language-07", "variant": 1, "text": "Il fait très beau près de la mer."}
+{"id": 144, "concept": "french_language", "pair_id": "french_language-08", "variant": 0, "text": "Elle prépare le dîner dans la cuisine."}
+{"id": 145, "concept": "french_language", "pair_id": "french_language-08", "variant": 1, "text": "Dans la cuisine, elle est en train de préparer le repas du soir."}
+{"id": 146, "concept": "french_language", "pair_id": "french_language-09", "variant": 0, "text": "Nous avons visité un musée pendant le week-end."}
+{"id": 147, "concept": "french_language", "pair_id": "french_language-09", "variant": 1, "text": "Ce week-end, nous sommes allés voir un musée."}
+{"id": 148, "concept": "french_language", "pair_id": "french_language-10", "variant": 0, "text": "La réunion commence à neuf heures précises."}
+{"id": 149, "concept": "french_language", "pair_id": "french_language-10", "variant": 1, "text": "Le rendez-vous débute exactement à neuf heures."}
+{"id": 150, "concept": "french_language", "pair_id": "french_language-11", "variant": 0, "text": "Mon ordinateur ne fonctionne plus correctement."}
+{"id": 151, "concept": "french_language", "pair_id": "french_language-11", "variant": 1, "text": "J’ai un problème : mon ordinateur marche mal maintenant."}
+{"id": 152, "concept": "french_language", "pair_id": "french_language-12", "variant": 0, "text": "Cette ville possède de nombreux bâtiments historiques."}
+{"id": 153, "concept": "french_language", "pair_id": "french_language-12", "variant": 1, "text": "On trouve beaucoup d’édifices historiques dans cette ville."}
+{"id": 154, "concept": "french_language", "pair_id": "french_language-13", "variant": 0, "text": "Il faut acheter du pain et des légumes."}
+{"id": 155, "concept": "french_language", "pair_id": "french_language-13", "variant": 1, "text": "Nous devons prendre du pain ainsi que des légumes."}
+{"id": 156, "concept": "french_language", "pair_id": "french_language-14", "variant": 0, "text": "Je cherche un appartement près de l’université."}
+{"id": 157, "concept": "french_language", "pair_id": "french_language-14", "variant": 1, "text": "Je voudrais trouver un logement proche de l’université."}
+{"id": 158, "concept": "french_language", "pair_id": "french_language-15", "variant": 0, "text": "Merci beaucoup pour votre aide."}
+{"id": 159, "concept": "french_language", "pair_id": "french_language-15", "variant": 1, "text": "Je vous remercie sincèrement de votre aide."}
+{"id": 160, "concept": "factual_entities", "pair_id": "factual_entities-00", "variant": 0, "text": "Tell me about Marie Curie and her scientific work."}
+{"id": 161, "concept": "factual_entities", "pair_id": "factual_entities-00", "variant": 1, "text": "Summarize the scientific contributions of Marie Curie."}
+{"id": 162, "concept": "factual_entities", "pair_id": "factual_entities-01", "variant": 0, "text": "What is notable about the city of Kyoto?"}
+{"id": 163, "concept": "factual_entities", "pair_id": "factual_entities-01", "variant": 1, "text": "Give a short factual overview of Kyoto."}
+{"id": 164, "concept": "factual_entities", "pair_id": "factual_entities-02", "variant": 0, "text": "Explain the role of the Nile in ancient Egypt."}
+{"id": 165, "concept": "factual_entities", "pair_id": "factual_entities-02", "variant": 1, "text": "Describe why the Nile mattered to ancient Egyptian civilization."}
+{"id": 166, "concept": "factual_entities", "pair_id": "factual_entities-03", "variant": 0, "text": "Who was Ada Lovelace?"}
+{"id": 167, "concept": "factual_entities", "pair_id": "factual_entities-03", "variant": 1, "text": "Provide a concise factual description of Ada Lovelace."}
+{"id": 168, "concept": "factual_entities", "pair_id": "factual_entities-04", "variant": 0, "text": "What is Mount Everest?"}
+{"id": 169, "concept": "factual_entities", "pair_id": "factual_entities-04", "variant": 1, "text": "Give basic factual information about Mount Everest."}
+{"id": 170, "concept": "factual_entities", "pair_id": "factual_entities-05", "variant": 0, "text": "Describe the planet Saturn."}
+{"id": 171, "concept": "factual_entities", "pair_id": "factual_entities-05", "variant": 1, "text": "Provide several factual details about Saturn."}
+{"id": 172, "concept": "factual_entities", "pair_id": "factual_entities-06", "variant": 0, "text": "What is the Great Barrier Reef?"}
+{"id": 173, "concept": "factual_entities", "pair_id": "factual_entities-06", "variant": 1, "text": "Give an overview of the Great Barrier Reef."}
+{"id": 174, "concept": "factual_entities", "pair_id": "factual_entities-07", "variant": 0, "text": "Tell me about Ludwig van Beethoven."}
+{"id": 175, "concept": "factual_entities", "pair_id": "factual_entities-07", "variant": 1, "text": "Summarize who Beethoven was and why he is remembered."}
+{"id": 176, "concept": "factual_entities", "pair_id": "factual_entities-08", "variant": 0, "text": "What is the Amazon River?"}
+{"id": 177, "concept": "factual_entities", "pair_id": "factual_entities-08", "variant": 1, "text": "Provide factual information about the Amazon River."}
+{"id": 178, "concept": "factual_entities", "pair_id": "factual_entities-09", "variant": 0, "text": "Explain what the Rosetta Stone is."}
+{"id": 179, "concept": "factual_entities", "pair_id": "factual_entities-09", "variant": 1, "text": "Describe the Rosetta Stone and its historical importance."}
+{"id": 180, "concept": "factual_entities", "pair_id": "factual_entities-10", "variant": 0, "text": "Tell me about the element gold."}
+{"id": 181, "concept": "factual_entities", "pair_id": "factual_entities-10", "variant": 1, "text": "Give a factual overview of the chemical element gold."}
+{"id": 182, "concept": "factual_entities", "pair_id": "factual_entities-11", "variant": 0, "text": "What was the Renaissance?"}
+{"id": 183, "concept": "factual_entities", "pair_id": "factual_entities-11", "variant": 1, "text": "Summarize the historical period known as the Renaissance."}
+{"id": 184, "concept": "factual_entities", "pair_id": "factual_entities-12", "variant": 0, "text": "Describe the city of Hamburg."}
+{"id": 185, "concept": "factual_entities", "pair_id": "factual_entities-12", "variant": 1, "text": "Give several basic facts about Hamburg, Germany."}
+{"id": 186, "concept": "factual_entities", "pair_id": "factual_entities-13", "variant": 0, "text": "Who was Alan Turing?"}
+{"id": 187, "concept": "factual_entities", "pair_id": "factual_entities-13", "variant": 1, "text": "Provide a factual summary of Alan Turing’s life and work."}
+{"id": 188, "concept": "factual_entities", "pair_id": "factual_entities-14", "variant": 0, "text": "What is the Pacific Ocean?"}
+{"id": 189, "concept": "factual_entities", "pair_id": "factual_entities-14", "variant": 1, "text": "Give a concise factual description of the Pacific Ocean."}
+{"id": 190, "concept": "factual_entities", "pair_id": "factual_entities-15", "variant": 0, "text": "Explain what DNA is."}
+{"id": 191, "concept": "factual_entities", "pair_id": "factual_entities-15", "variant": 1, "text": "Provide a factual description of DNA and its biological role."}
+{"id": 192, "concept": "uncertainty", "pair_id": "uncertainty-00", "variant": 0, "text": "I have not given enough information to know which box contains the key."}
+{"id": 193, "concept": "uncertainty", "pair_id": "uncertainty-00", "variant": 1, "text": "From the details provided, the location of the key cannot be determined."}
+{"id": 194, "concept": "uncertainty", "pair_id": "uncertainty-01", "variant": 0, "text": "The evidence is incomplete, so the cause remains uncertain."}
+{"id": 195, "concept": "uncertainty", "pair_id": "uncertainty-01", "variant": 1, "text": "There is insufficient evidence to identify the cause with confidence."}
+{"id": 196, "concept": "uncertainty", "pair_id": "uncertainty-02", "variant": 0, "text": "I do not know which route they chose because the text never says."}
+{"id": 197, "concept": "uncertainty", "pair_id": "uncertainty-02", "variant": 1, "text": "The passage does not specify the route, so the answer is unknown."}
+{"id": 198, "concept": "uncertainty", "pair_id": "uncertainty-03", "variant": 0, "text": "Without the missing measurement, the result cannot be calculated."}
+{"id": 199, "concept": "uncertainty", "pair_id": "uncertainty-03", "variant": 1, "text": "The calculation is underdetermined because a required value is absent."}
+{"id": 200, "concept": "uncertainty", "pair_id": "uncertainty-04", "variant": 0, "text": "The source does not state when the event happened."}
+{"id": 201, "concept": "uncertainty", "pair_id": "uncertainty-04", "variant": 1, "text": "The event date is not provided by the available source."}
+{"id": 202, "concept": "uncertainty", "pair_id": "uncertainty-05", "variant": 0, "text": "Several explanations fit the observations, so no single one is established."}
+{"id": 203, "concept": "uncertainty", "pair_id": "uncertainty-05", "variant": 1, "text": "The observations support multiple possibilities and do not settle on one explanation."}
+{"id": 204, "concept": "uncertainty", "pair_id": "uncertainty-06", "variant": 0, "text": "There is not enough context to identify who the pronoun refers to."}
+{"id": 205, "concept": "uncertainty", "pair_id": "uncertainty-06", "variant": 1, "text": "The pronoun’s referent is ambiguous given the limited context."}
+{"id": 206, "concept": "uncertainty", "pair_id": "uncertainty-07", "variant": 0, "text": "The sample is too small to draw a reliable conclusion."}
+{"id": 207, "concept": "uncertainty", "pair_id": "uncertainty-07", "variant": 1, "text": "A confident conclusion would be unjustified because the sample size is inadequate."}
+{"id": 208, "concept": "uncertainty", "pair_id": "uncertainty-08", "variant": 0, "text": "I cannot verify that claim from the information available."}
+{"id": 209, "concept": "uncertainty", "pair_id": "uncertainty-08", "variant": 1, "text": "The available information is insufficient to confirm the claim."}
+{"id": 210, "concept": "uncertainty", "pair_id": "uncertainty-09", "variant": 0, "text": "The instructions omit the final step, so the intended outcome is unclear."}
+{"id": 211, "concept": "uncertainty", "pair_id": "uncertainty-09", "variant": 1, "text": "Because the last instruction is missing, the desired result cannot be known."}
+{"id": 212, "concept": "uncertainty", "pair_id": "uncertainty-10", "variant": 0, "text": "We have two plausible answers and no evidence that distinguishes them."}
+{"id": 213, "concept": "uncertainty", "pair_id": "uncertainty-10", "variant": 1, "text": "Both answers remain possible because there is no discriminating evidence."}
+{"id": 214, "concept": "uncertainty", "pair_id": "uncertainty-11", "variant": 0, "text": "The report gives a range but not an exact value."}
+{"id": 215, "concept": "uncertainty", "pair_id": "uncertainty-11", "variant": 1, "text": "Only an interval is reported, so the precise value is unspecified."}
+{"id": 216, "concept": "uncertainty", "pair_id": "uncertainty-12", "variant": 0, "text": "The image is too blurry to read the number confidently."}
+{"id": 217, "concept": "uncertainty", "pair_id": "uncertainty-12", "variant": 1, "text": "The number cannot be identified reliably because the image lacks clarity."}
+{"id": 218, "concept": "uncertainty", "pair_id": "uncertainty-13", "variant": 0, "text": "No forecast was provided, so tomorrow’s value is unknown."}
+{"id": 219, "concept": "uncertainty", "pair_id": "uncertainty-13", "variant": 1, "text": "The future value cannot be stated because there is no forecast information."}
+{"id": 220, "concept": "uncertainty", "pair_id": "uncertainty-14", "variant": 0, "text": "The experiment was not repeated, so the finding remains tentative."}
+{"id": 221, "concept": "uncertainty", "pair_id": "uncertainty-14", "variant": 1, "text": "Without replication, the result should be treated as uncertain."}
+{"id": 222, "concept": "uncertainty", "pair_id": "uncertainty-15", "variant": 0, "text": "The text names several candidates but never identifies the winner."}
+{"id": 223, "concept": "uncertainty", "pair_id": "uncertainty-15", "variant": 1, "text": "A winner cannot be determined because the passage lists candidates without a result."}

docs/HF_DEPLOY.md

@@ -0,0 +1,24 @@
+# Hugging Face deployment
+
+FeatureLens targets a Gradio SDK Space with ZeroGPU hardware.
+
+## Why Gradio rather than Docker?
+
+ZeroGPU currently supports Gradio SDK Spaces. The workbench therefore uses `app.py` plus `requirements.txt` and avoids a Dockerfile.
+
+## Cold start
+
+On Hugging Face, `FEATURELENS_EAGER_LOAD` defaults to `1`. The runtime loads Qwen3-1.7B-Base and only the configured Qwen-Scope layer files. The default layer set is 4, 14 and 26.
+
+The model and SAEs are placed on `cuda` during startup. ZeroGPU provides CUDA emulation outside GPU-decorated callbacks and a real GPU during `spaces.GPU` execution.
+
+## GPU-decorated functions
+
+- feature inspection: 45-second maximum allocation;
+- baseline-vs-modified generation: 60-second maximum allocation.
+
+Keeping prompt length and generation length bounded protects the free daily quota and improves queue behavior.
+
+## No persistent benchmark jobs in the Space
+
+The full experiment is intentionally offline. A free ZeroGPU quota is appropriate for interactive inspection, not a hundreds-of-forward-passes benchmark. Run `python experiments/run_all.py` elsewhere once, commit the small report/catalog artifacts, and the Space will display them automatically.

docs/METHODOLOGY.md

@@ -0,0 +1,39 @@
+# Methodology
+
+## Primary hypothesis
+
+A sparse feature that is predictive of a semantic category is not automatically a causal mechanism for behavior. FeatureLens evaluates association and intervention separately.
+
+## Discovery data
+
+Seven controlled concepts are represented by 16 independently authored prompt pairs per concept. Each pair contains two paraphrases. The split is grouped by pair id so lexical near-duplicates never cross train and test.
+
+The final token residual is used for the offline concept benchmark because it has access to the complete prompt prefix and keeps the activation tensor small enough to reproduce easily. The live workbench is token-selectable and is not restricted to the final token.
+
+## Sparse feature evaluation
+
+For each configured layer, FeatureLens stores the TopK SAE code for each sample. Candidate features must fire at least three times on the training split. Features are ranked using training AUROC and an activation-rate contrast tie-break. AUROC and F1 shown in the report are then calculated on the held-out split.
+
+This makes the feature catalog a selection procedure rather than a post-hoc leaderboard over the test set.
+
+## Dense baseline
+
+A multinomial logistic-regression probe is fit to the dense residual vectors at the same layers. This asks whether the representation carries concept information even if no single SAE feature isolates it cleanly.
+
+## Causal intervention
+
+If the SAE code of the selected prompt residual is `z` and the selected feature is `i`, FeatureLens changes only coefficient `z_i`. The resulting decoder delta is added back to the original residual. Full SAE reconstruction is never substituted for the original activation.
+
+Ablation uses `Δz_i = -z_i`; 2× amplification uses `Δz_i = z_i`.
+
+## Negative control
+
+For every SAE residual perturbation, a deterministic random direction with identical L2 norm is generated and patched at the same layer/token. This does not prove specificity by itself, but it gives a much stronger baseline than reporting an intervention effect without a perturbation control.
+
+## Causal outcomes
+
+The causal task set is separate from feature discovery. The main behavioral score is the change in log-probability of the first token of a target completion. The pipeline also stores probability delta, target rank change, JS divergence of the next-token distribution, and whether the top-1 token changed.
+
+## Interpretation
+
+The auto-generated report uses conservative heuristics to summarize measured results, but preserves all raw rows. A high predictive AUROC with a small causal effect is explicitly reported as evidence that correlation did not translate into strong downstream control.

experiments/build_dataset.py

@@ -0,0 +1,200 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+DATA_DIR = ROOT / 'data'
+
+PAIRS: dict[str, list[tuple[str, str]]] = {
+    'code': [
+        ('Write a Python function that returns the larger of two integers.', 'In Python, create a function that chooses the maximum of two integer inputs.'),
+        ('Explain what a recursive function does in a program.', 'Describe recursion in the context of a programming function.'),
+        ('Show a JavaScript loop that prints numbers from one to five.', 'Give JavaScript code for iterating over the integers 1 through 5 and logging them.'),
+        ('What is the purpose of a hash map in software?', 'Explain why programmers use hash maps or dictionaries.'),
+        ('Write SQL that selects every row from a table named users.', 'Provide a SQL query to retrieve all records in the users table.'),
+        ('How do I catch an exception in Python?', 'Show the Python syntax for handling an exception with try and except.'),
+        ('Explain the difference between a list and a tuple in Python.', 'Compare Python lists with tuples.'),
+        ('Write a function that checks whether a string is a palindrome.', 'Create code that tests if text reads the same forwards and backwards.'),
+        ('What does git rebase do?', 'Describe the effect of rebasing a Git branch.'),
+        ('How can a REST API return JSON?', 'Explain how a web API endpoint sends a JSON response.'),
+        ('Write pseudocode for binary search.', 'Describe the binary-search algorithm as pseudocode.'),
+        ('What is a class constructor used for?', 'Explain the role of a constructor when creating an object.'),
+        ('Give a regular expression that matches a basic email-like string.', 'Show a regex pattern for a simple email address format.'),
+        ('Why is time complexity important when comparing algorithms?', 'Explain why Big-O runtime matters in algorithm analysis.'),
+        ('How do unit tests help a codebase?', 'Describe the purpose of automated unit testing in software.'),
+        ('Write Python that sorts a list of dictionaries by an age key.', 'Show how to order Python dictionaries in a list according to their age field.'),
+    ],
+    'mathematics': [
+        ('Find the derivative of x cubed plus two x.', 'Differentiate the function x^3 + 2x.'),
+        ('Solve the equation 3x + 5 = 20.', 'Find x when three x plus five equals twenty.'),
+        ('What is the integral of cosine x?', 'Compute an antiderivative of cos(x).'),
+        ('Factor the polynomial x squared minus nine.', 'Rewrite x^2 - 9 as a product of factors.'),
+        ('Calculate the mean of 4, 7, 9, and 10.', 'What is the arithmetic average of 4, 7, 9, 10?'),
+        ('Explain the Pythagorean theorem.', 'State the relationship among the sides of a right triangle.'),
+        ('What is the determinant of a two by two matrix?', 'Give the formula for the determinant of a 2x2 matrix.'),
+        ('Simplify two to the third power times two squared.', 'Reduce 2^3 multiplied by 2^2 using exponent rules.'),
+        ('Convert one half into a percentage.', 'Express 1/2 as a percent.'),
+        ('What is the probability of heads on a fair coin?', 'For a fair coin, calculate the chance of flipping heads.'),
+        ('Solve x squared equals sixteen.', 'Find the real values of x satisfying x^2 = 16.'),
+        ('What is the slope between points (1,2) and (3,6)?', 'Calculate the gradient of the line through (1,2) and (3,6).'),
+        ('Explain what a prime number is.', 'Define a prime integer.'),
+        ('Compute the dot product of vectors (1,2) and (3,4).', 'Find (1,2) · (3,4).'),
+        ('What is log base ten of one thousand?', 'Evaluate log_10(1000).'),
+        ('A circle has radius three. What is its area?', 'Calculate the area of a circle whose radius is 3.'),
+    ],
+    'positive_sentiment': [
+        ('The film was delightful, clever, and beautifully acted.', 'I found the movie charming, smart, and wonderfully performed.'),
+        ('This restaurant served an excellent meal and the staff were kind.', 'The food was fantastic and the service team was genuinely friendly.'),
+        ('I am very pleased with how reliable this laptop has been.', 'This laptop has worked dependably and I am extremely satisfied with it.'),
+        ('The concert was energetic and unforgettable.', 'I had an amazing time at the lively, memorable concert.'),
+        ('Her explanation was clear and genuinely helpful.', 'She explained the topic in a useful and easy-to-understand way.'),
+        ('The hotel room was spotless and comfortable.', 'Our room was exceptionally clean and pleasant to stay in.'),
+        ('I loved the book from beginning to end.', 'The novel kept me delighted all the way through.'),
+        ('The new update makes the app much easier to use.', 'After the update, the application feels significantly more convenient.'),
+        ('Their customer support solved my problem quickly.', 'Support handled the issue fast and left me very happy.'),
+        ('The hike had spectacular views and perfect weather.', 'We enjoyed gorgeous scenery and wonderful conditions on the hike.'),
+        ('The presentation was engaging and well organized.', 'I enjoyed the talk because it was compelling and structured clearly.'),
+        ('This camera takes sharp photos and feels great to use.', 'The camera produces crisp images and has a satisfying design.'),
+        ('The workshop exceeded my expectations.', 'I was impressed because the workshop was even better than I expected.'),
+        ('Dinner turned out wonderfully and everyone enjoyed it.', 'The evening meal was a success and all of us had a great time.'),
+        ('The museum exhibition was fascinating.', 'I thought the exhibition was deeply interesting and rewarding.'),
+        ('The train journey was smooth and relaxing.', 'The trip by train felt easy, calm, and comfortable.'),
+    ],
+    'negative_sentiment': [
+        ('The film was tedious, confusing, and badly acted.', 'I found the movie boring, incoherent, and poorly performed.'),
+        ('This restaurant served cold food and the staff were rude.', 'The meal arrived cold and the service team behaved unpleasantly.'),
+        ('I am disappointed by how unreliable this laptop has been.', 'This laptop keeps failing and I am extremely dissatisfied with it.'),
+        ('The concert was chaotic and forgettable.', 'I had a miserable time at the disorganized, dull concert.'),
+        ('Her explanation was unclear and unhelpful.', 'She explained the topic in a confusing way that did not help me.'),
+        ('The hotel room was dirty and uncomfortable.', 'Our room was unpleasant, unclean, and difficult to relax in.'),
+        ('I regretted reading the book.', 'The novel was a frustrating waste of my time.'),
+        ('The new update makes the app harder to use.', 'After the update, the application feels significantly more awkward.'),
+        ('Their customer support ignored my problem.', 'Support failed to resolve the issue and left me angry.'),
+        ('The hike had awful weather and disappointing views.', 'We dealt with terrible conditions and underwhelming scenery on the hike.'),
+        ('The presentation was dull and poorly organized.', 'I disliked the talk because it was tedious and structured badly.'),
+        ('This camera takes blurry photos and feels cheap.', 'The camera produces soft images and has a flimsy design.'),
+        ('The workshop fell far below my expectations.', 'I was disappointed because the workshop was much worse than I expected.'),
+        ('Dinner went badly and nobody enjoyed it.', 'The evening meal was a failure and all of us had a poor time.'),
+        ('The museum exhibition was painfully boring.', 'I thought the exhibition was dull and unrewarding.'),
+        ('The train journey was stressful and uncomfortable.', 'The trip by train felt frustrating, noisy, and unpleasant.'),
+    ],
+    'french_language': [
+        ('Bonjour, comment allez-vous aujourd’hui ?', 'Salut, comment vas-tu aujourd’hui ?'),
+        ('Je voudrais réserver une table pour deux personnes.', 'Puis-je réserver une table pour deux, s’il vous plaît ?'),
+        ('La bibliothèque ferme à dix-huit heures.', 'La bibliothèque est ouverte jusqu’à dix-huit heures.'),
+        ('Ce livre raconte une histoire très intéressante.', 'L’histoire racontée dans ce livre est vraiment intéressante.'),
+        ('Nous allons prendre le train demain matin.', 'Demain matin, nous voyagerons en train.'),
+        ('Pouvez-vous m’indiquer le chemin vers la gare ?', 'Comment puis-je aller jusqu’à la gare ?'),
+        ('J’aime apprendre de nouvelles langues.', 'Apprendre des langues nouvelles me plaît beaucoup.'),
+        ('Le temps est magnifique au bord de la mer.', 'Il fait très beau près de la mer.'),
+        ('Elle prépare le dîner dans la cuisine.', 'Dans la cuisine, elle est en train de préparer le repas du soir.'),
+        ('Nous avons visité un musée pendant le week-end.', 'Ce week-end, nous sommes allés voir un musée.'),
+        ('La réunion commence à neuf heures précises.', 'Le rendez-vous débute exactement à neuf heures.'),
+        ('Mon ordinateur ne fonctionne plus correctement.', 'J’ai un problème : mon ordinateur marche mal maintenant.'),
+        ('Cette ville possède de nombreux bâtiments historiques.', 'On trouve beaucoup d’édifices historiques dans cette ville.'),
+        ('Il faut acheter du pain et des légumes.', 'Nous devons prendre du pain ainsi que des légumes.'),
+        ('Je cherche un appartement près de l’université.', 'Je voudrais trouver un logement proche de l’université.'),
+        ('Merci beaucoup pour votre aide.', 'Je vous remercie sincèrement de votre aide.'),
+    ],
+    'factual_entities': [
+        ('Tell me about Marie Curie and her scientific work.', 'Summarize the scientific contributions of Marie Curie.'),
+        ('What is notable about the city of Kyoto?', 'Give a short factual overview of Kyoto.'),
+        ('Explain the role of the Nile in ancient Egypt.', 'Describe why the Nile mattered to ancient Egyptian civilization.'),
+        ('Who was Ada Lovelace?', 'Provide a concise factual description of Ada Lovelace.'),
+        ('What is Mount Everest?', 'Give basic factual information about Mount Everest.'),
+        ('Describe the planet Saturn.', 'Provide several factual details about Saturn.'),
+        ('What is the Great Barrier Reef?', 'Give an overview of the Great Barrier Reef.'),
+        ('Tell me about Ludwig van Beethoven.', 'Summarize who Beethoven was and why he is remembered.'),
+        ('What is the Amazon River?', 'Provide factual information about the Amazon River.'),
+        ('Explain what the Rosetta Stone is.', 'Describe the Rosetta Stone and its historical importance.'),
+        ('Tell me about the element gold.', 'Give a factual overview of the chemical element gold.'),
+        ('What was the Renaissance?', 'Summarize the historical period known as the Renaissance.'),
+        ('Describe the city of Hamburg.', 'Give several basic facts about Hamburg, Germany.'),
+        ('Who was Alan Turing?', 'Provide a factual summary of Alan Turing’s life and work.'),
+        ('What is the Pacific Ocean?', 'Give a concise factual description of the Pacific Ocean.'),
+        ('Explain what DNA is.', 'Provide a factual description of DNA and its biological role.'),
+    ],
+    'uncertainty': [
+        ('I have not given enough information to know which box contains the key.', 'From the details provided, the location of the key cannot be determined.'),
+        ('The evidence is incomplete, so the cause remains uncertain.', 'There is insufficient evidence to identify the cause with confidence.'),
+        ('I do not know which route they chose because the text never says.', 'The passage does not specify the route, so the answer is unknown.'),
+        ('Without the missing measurement, the result cannot be calculated.', 'The calculation is underdetermined because a required value is absent.'),
+        ('The source does not state when the event happened.', 'The event date is not provided by the available source.'),
+        ('Several explanations fit the observations, so no single one is established.', 'The observations support multiple possibilities and do not settle on one explanation.'),
+        ('There is not enough context to identify who the pronoun refers to.', 'The pronoun’s referent is ambiguous given the limited context.'),
+        ('The sample is too small to draw a reliable conclusion.', 'A confident conclusion would be unjustified because the sample size is inadequate.'),
+        ('I cannot verify that claim from the information available.', 'The available information is insufficient to confirm the claim.'),
+        ('The instructions omit the final step, so the intended outcome is unclear.', 'Because the last instruction is missing, the desired result cannot be known.'),
+        ('We have two plausible answers and no evidence that distinguishes them.', 'Both answers remain possible because there is no discriminating evidence.'),
+        ('The report gives a range but not an exact value.', 'Only an interval is reported, so the precise value is unspecified.'),
+        ('The image is too blurry to read the number confidently.', 'The number cannot be identified reliably because the image lacks clarity.'),
+        ('No forecast was provided, so tomorrow’s value is unknown.', 'The future value cannot be stated because there is no forecast information.'),
+        ('The experiment was not repeated, so the finding remains tentative.', 'Without replication, the result should be treated as uncertain.'),
+        ('The text names several candidates but never identifies the winner.', 'A winner cannot be determined because the passage lists candidates without a result.'),
+    ],
+}
+
+CAUSAL_TASKS = [
+    {'concept': 'positive_sentiment', 'prompt': 'Sentiment: The meal was wonderful and the staff were kind. Label:', 'target': ' positive'},
+    {'concept': 'positive_sentiment', 'prompt': 'Sentiment: I loved the thoughtful, beautifully written novel. Label:', 'target': ' positive'},
+    {'concept': 'positive_sentiment', 'prompt': 'Sentiment: The service was fast and extremely helpful. Label:', 'target': ' positive'},
+    {'concept': 'positive_sentiment', 'prompt': 'Sentiment: The journey was comfortable and enjoyable. Label:', 'target': ' positive'},
+    {'concept': 'negative_sentiment', 'prompt': 'Sentiment: The meal was awful and the staff were rude. Label:', 'target': ' negative'},
+    {'concept': 'negative_sentiment', 'prompt': 'Sentiment: I hated the tedious, badly written novel. Label:', 'target': ' negative'},
+    {'concept': 'negative_sentiment', 'prompt': 'Sentiment: The service was slow and completely unhelpful. Label:', 'target': ' negative'},
+    {'concept': 'negative_sentiment', 'prompt': 'Sentiment: The journey was stressful and unpleasant. Label:', 'target': ' negative'},
+    {'concept': 'mathematics', 'prompt': '2 + 3 =', 'target': ' 5'},
+    {'concept': 'mathematics', 'prompt': '7 - 4 =', 'target': ' 3'},
+    {'concept': 'mathematics', 'prompt': '6 * 2 =', 'target': ' 12'},
+    {'concept': 'mathematics', 'prompt': 'The square root of 81 is', 'target': ' 9'},
+    {'concept': 'code', 'prompt': 'Python function declaration keyword:', 'target': ' def'},
+    {'concept': 'code', 'prompt': 'In Python, an exception handler begins with the keyword', 'target': ' except'},
+    {'concept': 'code', 'prompt': 'SQL keyword used to retrieve rows:', 'target': ' SELECT'},
+    {'concept': 'code', 'prompt': 'A Python conditional branch commonly starts with', 'target': ' if'},
+    {'concept': 'french_language', 'prompt': "Translate 'hello' into French:", 'target': ' bonjour'},
+    {'concept': 'french_language', 'prompt': "Translate 'thank you' into French:", 'target': ' merci'},
+    {'concept': 'french_language', 'prompt': "Translate 'yes' into French:", 'target': ' oui'},
+    {'concept': 'french_language', 'prompt': "Translate 'good evening' into French:", 'target': ' bonsoir'},
+    {'concept': 'factual_entities', 'prompt': 'The scientist associated with radium research, Marie', 'target': ' Curie'},
+    {'concept': 'factual_entities', 'prompt': 'The composer of the Fifth Symphony, Ludwig van', 'target': ' Beethoven'},
+    {'concept': 'factual_entities', 'prompt': 'The computer scientist known for the Turing machine, Alan', 'target': ' Turing'},
+    {'concept': 'factual_entities', 'prompt': 'The Japanese city famous for many historic temples,', 'target': ' Kyoto'},
+    {'concept': 'uncertainty', 'prompt': 'The evidence is insufficient. The answer is', 'target': ' unknown'},
+    {'concept': 'uncertainty', 'prompt': 'The passage never states the value, so it is', 'target': ' unknown'},
+    {'concept': 'uncertainty', 'prompt': 'There is not enough information to determine the result. It remains', 'target': ' uncertain'},
+    {'concept': 'uncertainty', 'prompt': 'No reliable conclusion can be drawn; the outcome is', 'target': ' unclear'},
+]
+
+
+def main() -> None:
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    prompt_path = DATA_DIR / 'prompts.jsonl'
+    with prompt_path.open('w', encoding='utf-8') as handle:
+        sample_id = 0
+        for concept, pairs in PAIRS.items():
+            for pair_num, (a, b) in enumerate(pairs):
+                pair_id = f'{concept}-{pair_num:02d}'
+                for variant, text in enumerate((a, b)):
+                    record = {
+                        'id': sample_id,
+                        'concept': concept,
+                        'pair_id': pair_id,
+                        'variant': variant,
+                        'text': text,
+                    }
+                    handle.write(json.dumps(record, ensure_ascii=False) + '\n')
+                    sample_id += 1
+
+    causal_path = DATA_DIR / 'causal_tasks.jsonl'
+    with causal_path.open('w', encoding='utf-8') as handle:
+        for idx, task in enumerate(CAUSAL_TASKS):
+            record = {'id': idx, **task}
+            handle.write(json.dumps(record, ensure_ascii=False) + '\n')
+
+    print(f'Wrote {sample_id} discovery prompts to {prompt_path}')
+    print(f'Wrote {len(CAUSAL_TASKS)} causal tasks to {causal_path}')
+
+
+if __name__ == '__main__':
+    main()

experiments/collect_activations.py

@@ -0,0 +1,151 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import numpy as np
+import scipy.sparse as sp
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from experiments.common import ARTIFACT_DIR, DATA_DIR, load_jsonl, set_seed
+from featurelens.config import SETTINGS
+from featurelens.metrics import reconstruction_metrics
+from featurelens.sae import SAEStore
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Collect residual and Qwen-Scope SAE activations.')
+    parser.add_argument('--input', type=Path, default=DATA_DIR / 'prompts.jsonl')
+    parser.add_argument('--output-dir', type=Path, default=ARTIFACT_DIR / 'activations')
+    parser.add_argument('--batch-size', type=int, default=16)
+    parser.add_argument('--max-length', type=int, default=192)
+    parser.add_argument('--seed', type=int, default=42)
+    parser.add_argument('--layers', type=int, nargs='+', default=list(SETTINGS.layers))
+    return parser.parse_args()
+
+
+def _build_sparse(encodings: list, n_rows: int, width: int) -> sp.csr_matrix:
+    row_ids: list[int] = []
+    col_ids: list[int] = []
+    values: list[float] = []
+    for row, encoding in enumerate(encodings):
+        idx = encoding.indices.detach().cpu().numpy().reshape(-1)
+        vals = encoding.values.detach().float().cpu().numpy().reshape(-1)
+        positive = vals > 0
+        row_ids.extend([row] * int(positive.sum()))
+        col_ids.extend(idx[positive].astype(int).tolist())
+        values.extend(vals[positive].astype(float).tolist())
+    return sp.csr_matrix((values, (row_ids, col_ids)), shape=(n_rows, width), dtype=np.float32)
+
+
+@torch.inference_mode()
+def main() -> None:
+    args = parse_args()
+    set_seed(args.seed)
+    rows = load_jsonl(args.input)
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    model_dtype = torch.float16 if device.type == 'cuda' else torch.float32
+    tokenizer = AutoTokenizer.from_pretrained(SETTINGS.model_id)
+    if tokenizer.pad_token_id is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    tokenizer.padding_side = 'left'
+    model = AutoModelForCausalLM.from_pretrained(
+        SETTINGS.model_id,
+        torch_dtype=model_dtype,
+        low_cpu_mem_usage=True,
+    ).to(device)
+    model.eval()
+    sae_store = SAEStore(
+        SETTINGS.sae_repo_id,
+        layers=args.layers,
+        device=device,
+        dtype=torch.float32,
+        top_k=SETTINGS.sae_top_k,
+    )
+
+    residuals: dict[int, list[np.ndarray]] = {layer: [] for layer in args.layers}
+    encodings: dict[int, list] = {layer: [] for layer in args.layers}
+    recon_stats: dict[int, list[dict[str, float]]] = {layer: [] for layer in args.layers}
+
+    for start in range(0, len(rows), args.batch_size):
+        batch_rows = rows[start : start + args.batch_size]
+        texts = [row['text'] for row in batch_rows]
+        batch = tokenizer(
+            texts,
+            return_tensors='pt',
+            padding=True,
+            truncation=True,
+            max_length=args.max_length,
+        )
+        batch = {key: value.to(device) for key, value in batch.items()}
+        captured: dict[int, torch.Tensor] = {}
+        handles = []
+
+        def make_hook(layer: int):
+            def hook(_module, _inputs, output):
+                hidden = output[0] if isinstance(output, tuple) else output
+                captured[layer] = hidden.detach()
+
+            return hook
+
+        for layer in args.layers:
+            handles.append(model.model.layers[layer].register_forward_hook(make_hook(layer)))
+        model(**batch, use_cache=False)
+        for handle in handles:
+            handle.remove()
+
+        for layer in args.layers:
+            sae = sae_store.get(layer)
+            final_token_residuals = captured[layer][:, -1, :]
+            batch_encoding = sae.encode(final_token_residuals)
+            for row_idx in range(final_token_residuals.shape[0]):
+                residual = final_token_residuals[row_idx]
+                from featurelens.sae import SparseEncoding
+
+                encoding = SparseEncoding(
+                    indices=batch_encoding.indices[row_idx],
+                    values=batch_encoding.values[row_idx],
+                )
+                reconstructed = sae.decode_sparse(encoding)
+                residuals[layer].append(residual.detach().float().cpu().numpy().astype(np.float16))
+                encodings[layer].append(encoding)
+                recon_stats[layer].append(reconstruction_metrics(residual, reconstructed))
+
+        print(f'Processed {min(start + args.batch_size, len(rows))}/{len(rows)} prompts', flush=True)
+
+    for layer in args.layers:
+        residual_array = np.stack(residuals[layer], axis=0)
+        np.save(args.output_dir / f'residuals_layer{layer}.npy', residual_array)
+        sparse = _build_sparse(encodings[layer], len(rows), SETTINGS.sae_width)
+        sp.save_npz(args.output_dir / f'features_layer{layer}.npz', sparse, compressed=True)
+        summary = {
+            'layer': layer,
+            'n_samples': len(rows),
+            'mean_cosine': float(np.mean([x['cosine'] for x in recon_stats[layer]])),
+            'mean_nmse': float(np.mean([x['nmse'] for x in recon_stats[layer]])),
+            'median_nmse': float(np.median([x['nmse'] for x in recon_stats[layer]])),
+            'mean_active_features': float(np.mean(np.diff(sparse.indptr))),
+        }
+        (args.output_dir / f'reconstruction_layer{layer}.json').write_text(
+            json.dumps(summary, indent=2), encoding='utf-8'
+        )
+
+    metadata = {
+        'model_id': SETTINGS.model_id,
+        'sae_repo_id': SETTINGS.sae_repo_id,
+        'layers': args.layers,
+        'top_k': SETTINGS.sae_top_k,
+        'width': SETTINGS.sae_width,
+        'n_samples': len(rows),
+        'rows': rows,
+    }
+    (args.output_dir / 'metadata.json').write_text(json.dumps(metadata, indent=2), encoding='utf-8')
+    print(f'Activation artifacts written to {args.output_dir}')
+
+
+if __name__ == '__main__':
+    main()

experiments/common.py

@@ -0,0 +1,30 @@
+from __future__ import annotations
+
+import json
+import random
+from pathlib import Path
+
+import numpy as np
+import torch
+
+ROOT = Path(__file__).resolve().parents[1]
+DATA_DIR = ROOT / 'data'
+ARTIFACT_DIR = ROOT / 'artifacts'
+
+
+def load_jsonl(path: str | Path) -> list[dict]:
+    rows = []
+    with Path(path).open(encoding='utf-8') as handle:
+        for line in handle:
+            line = line.strip()
+            if line:
+                rows.append(json.loads(line))
+    return rows
+
+
+def set_seed(seed: int) -> None:
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    if torch.cuda.is_available():
+        torch.cuda.manual_seed_all(seed)

experiments/evaluate_features.py

@@ -0,0 +1,204 @@
+from __future__ import annotations
+
+import argparse
+import csv
+import json
+from collections import defaultdict
+from pathlib import Path
+
+import numpy as np
+import scipy.sparse as sp
+from sklearn.linear_model import LogisticRegression
+from sklearn.metrics import f1_score, precision_recall_curve, roc_auc_score
+from sklearn.pipeline import make_pipeline
+from sklearn.preprocessing import LabelEncoder, StandardScaler
+
+from experiments.common import ARTIFACT_DIR
+from experiments.split import grouped_concept_split
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Evaluate predictive SAE features and residual probes.')
+    parser.add_argument('--activation-dir', type=Path, default=ARTIFACT_DIR / 'activations')
+    parser.add_argument('--output-dir', type=Path, default=ARTIFACT_DIR)
+    parser.add_argument('--top-features', type=int, default=20)
+    parser.add_argument('--min-train-fires', type=int, default=3)
+    parser.add_argument('--seed', type=int, default=42)
+    return parser.parse_args()
+
+
+def _best_threshold(y_true: np.ndarray, scores: np.ndarray) -> float:
+    precision, recall, thresholds = precision_recall_curve(y_true, scores)
+    if thresholds.size == 0:
+        return 0.0
+    denom = precision[:-1] + recall[:-1]
+    f1 = np.divide(
+        2 * precision[:-1] * recall[:-1],
+        denom,
+        out=np.zeros_like(denom),
+        where=denom > 0,
+    )
+    return float(thresholds[int(np.argmax(f1))])
+
+
+def _evaluate_feature(
+    train_scores: np.ndarray,
+    test_scores: np.ndarray,
+    y_train: np.ndarray,
+    y_test: np.ndarray,
+) -> tuple[float, float, float]:
+    threshold = _best_threshold(y_train, train_scores)
+    train_auc = float(roc_auc_score(y_train, train_scores))
+    test_auc = float(roc_auc_score(y_test, test_scores))
+    pred = (test_scores >= threshold).astype(int)
+    f1 = float(f1_score(y_test, pred, zero_division=0))
+    return train_auc, test_auc, f1, threshold
+
+
+def _sparse_cosine(a: sp.csr_matrix, b: sp.csr_matrix) -> float:
+    numerator = float(a.multiply(b).sum())
+    denom = float(np.sqrt(a.multiply(a).sum()) * np.sqrt(b.multiply(b).sum()))
+    return numerator / denom if denom > 0 else 1.0
+
+
+def _jaccard(a: sp.csr_matrix, b: sp.csr_matrix) -> float:
+    sa = set(a.indices.tolist())
+    sb = set(b.indices.tolist())
+    union = sa | sb
+    return len(sa & sb) / len(union) if union else 1.0
+
+
+def main() -> None:
+    args = parse_args()
+    args.output_dir.mkdir(parents=True, exist_ok=True)
+    metadata = json.loads((args.activation_dir / 'metadata.json').read_text(encoding='utf-8'))
+    rows = metadata['rows']
+    layers = [int(x) for x in metadata['layers']]
+    train_idx, test_idx = grouped_concept_split(rows, seed=args.seed)
+    labels = np.array([row['concept'] for row in rows])
+    concepts = sorted(set(labels.tolist()))
+
+    feature_rows: list[dict] = []
+    layer_rows: list[dict] = []
+    stability_rows: list[dict] = []
+
+    encoder = LabelEncoder().fit(labels)
+    y_all = encoder.transform(labels)
+    y_train_multi = y_all[train_idx]
+    y_test_multi = y_all[test_idx]
+
+    for layer in layers:
+        x = sp.load_npz(args.activation_dir / f'features_layer{layer}.npz').tocsr()
+        x_csc = x.tocsc()
+        residuals = np.load(args.activation_dir / f'residuals_layer{layer}.npy').astype(np.float32)
+        recon = json.loads(
+            (args.activation_dir / f'reconstruction_layer{layer}.json').read_text(encoding='utf-8')
+        )
+
+        probe = make_pipeline(
+            StandardScaler(),
+            LogisticRegression(max_iter=2500, class_weight='balanced', random_state=args.seed),
+        )
+        probe.fit(residuals[train_idx], y_train_multi)
+        pred = probe.predict(residuals[test_idx])
+        probs = probe.predict_proba(residuals[test_idx])
+        probe_f1 = float(f1_score(y_test_multi, pred, average='macro'))
+        probe_auc = float(
+            roc_auc_score(y_test_multi, probs, multi_class='ovr', average='macro')
+        )
+
+        layer_rows.append(
+            {
+                'layer': layer,
+                'linear_probe_macro_auroc': probe_auc,
+                'linear_probe_macro_f1': probe_f1,
+                'reconstruction_cosine': recon['mean_cosine'],
+                'reconstruction_nmse': recon['mean_nmse'],
+                'mean_active_features': recon['mean_active_features'],
+            }
+        )
+
+        train_matrix = x[train_idx]
+        candidate_ids, counts = np.unique(train_matrix.indices, return_counts=True)
+        candidate_ids = candidate_ids[counts >= args.min_train_fires]
+
+        for concept in concepts:
+            y_train = (labels[train_idx] == concept).astype(int)
+            y_test = (labels[test_idx] == concept).astype(int)
+            concept_results: list[dict] = []
+            for feature_id in candidate_ids.tolist():
+                train_scores = x_csc[train_idx, feature_id].toarray().ravel()
+                if int((train_scores > 0).sum()) < args.min_train_fires:
+                    continue
+                test_scores = x_csc[test_idx, feature_id].toarray().ravel()
+                train_auc, test_auc, f1, threshold = _evaluate_feature(
+                    train_scores, test_scores, y_train, y_test
+                )
+                pos_train = train_scores[y_train == 1]
+                neg_train = train_scores[y_train == 0]
+                result = {
+                    'layer': layer,
+                    'concept': concept,
+                    'feature_id': int(feature_id),
+                    'train_auroc': train_auc,
+                    'auroc': test_auc,
+                    'f1': f1,
+                    'threshold': threshold,
+                    'activation_rate_pos': float(np.mean(pos_train > 0)),
+                    'activation_rate_neg': float(np.mean(neg_train > 0)),
+                    'mean_activation_pos': float(np.mean(pos_train)),
+                    'mean_activation_neg': float(np.mean(neg_train)),
+                }
+                concept_results.append(result)
+            concept_results.sort(
+                key=lambda item: (item['train_auroc'], item['activation_rate_pos'] - item['activation_rate_neg']),
+                reverse=True,
+            )
+            feature_rows.extend(concept_results[: args.top_features])
+
+        pair_map: dict[str, list[int]] = defaultdict(list)
+        for idx, row in enumerate(rows):
+            pair_map[row['pair_id']].append(idx)
+        for pair_id, indices in pair_map.items():
+            if len(indices) != 2:
+                continue
+            a, b = indices
+            stability_rows.append(
+                {
+                    'layer': layer,
+                    'pair_id': pair_id,
+                    'concept': rows[a]['concept'],
+                    'topk_jaccard': _jaccard(x.getrow(a), x.getrow(b)),
+                    'sparse_cosine': _sparse_cosine(x.getrow(a), x.getrow(b)),
+                }
+            )
+        print(f'Evaluated layer {layer}', flush=True)
+
+    with (args.output_dir / 'feature_catalog.csv').open('w', newline='', encoding='utf-8') as handle:
+        writer = csv.DictWriter(handle, fieldnames=list(feature_rows[0].keys()))
+        writer.writeheader()
+        writer.writerows(feature_rows)
+
+    with (args.output_dir / 'layer_metrics.csv').open('w', newline='', encoding='utf-8') as handle:
+        writer = csv.DictWriter(handle, fieldnames=list(layer_rows[0].keys()))
+        writer.writeheader()
+        writer.writerows(layer_rows)
+
+    with (args.output_dir / 'stability.csv').open('w', newline='', encoding='utf-8') as handle:
+        writer = csv.DictWriter(handle, fieldnames=list(stability_rows[0].keys()))
+        writer.writeheader()
+        writer.writerows(stability_rows)
+
+    split_payload = {
+        'seed': args.seed,
+        'train_indices': train_idx,
+        'test_indices': test_idx,
+        'n_train': len(train_idx),
+        'n_test': len(test_idx),
+    }
+    (args.output_dir / 'split.json').write_text(json.dumps(split_payload, indent=2), encoding='utf-8')
+    print(f'Wrote evaluation artifacts to {args.output_dir}')
+
+
+if __name__ == '__main__':
+    main()

experiments/make_report.py

@@ -0,0 +1,209 @@
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+from experiments.common import ARTIFACT_DIR
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Build a truthful experiment report from saved metrics.')
+    parser.add_argument('--artifact-dir', type=Path, default=ARTIFACT_DIR)
+    return parser.parse_args()
+
+
+def _selected_features(catalog: pd.DataFrame) -> pd.DataFrame:
+    scored = catalog.copy()
+    scored['activation_contrast'] = (
+        scored['activation_rate_pos'] - scored['activation_rate_neg']
+    )
+    ordered = scored.sort_values(
+        ['concept', 'train_auroc', 'activation_contrast'],
+        ascending=[True, False, False],
+    )
+    return ordered.groupby('concept', as_index=False).first()
+
+
+def _save_plots(artifact_dir: Path, selected: pd.DataFrame, layers: pd.DataFrame, causal: pd.DataFrame) -> None:
+    fig_dir = artifact_dir / 'figures'
+    fig_dir.mkdir(parents=True, exist_ok=True)
+
+    figure = plt.figure(figsize=(7.5, 4.2))
+    ax = figure.add_subplot(111)
+    ordered = selected.sort_values('auroc')
+    ax.barh(ordered['concept'], ordered['auroc'])
+    ax.axvline(0.5, linewidth=1, linestyle='--')
+    ax.set_xlabel('Held-out AUROC')
+    ax.set_title('Selected SAE feature predictiveness')
+    figure.tight_layout()
+    figure.savefig(fig_dir / 'feature_auroc.png', dpi=160)
+    plt.close(figure)
+
+    figure = plt.figure(figsize=(7.0, 4.2))
+    ax = figure.add_subplot(111)
+    ax.plot(layers['layer'], layers['linear_probe_macro_auroc'], marker='o', label='Linear probe AUROC')
+    ax.plot(layers['layer'], layers['reconstruction_cosine'], marker='o', label='SAE reconstruction cosine')
+    ax.set_xlabel('Layer')
+    ax.set_ylim(0, 1.05)
+    ax.set_title('Layer-wise representation diagnostics')
+    ax.legend()
+    figure.tight_layout()
+    figure.savefig(fig_dir / 'layer_diagnostics.png', dpi=160)
+    plt.close(figure)
+
+    grouped = (
+        causal.groupby(['intervention', 'condition'])['target_logprob_delta']
+        .apply(lambda values: float(np.mean(np.abs(values))))
+        .reset_index(name='mean_abs_delta_logp')
+    )
+    pivot = grouped.pivot(index='intervention', columns='condition', values='mean_abs_delta_logp')
+    figure = plt.figure(figsize=(7.0, 4.2))
+    ax = figure.add_subplot(111)
+    pivot.plot(kind='bar', ax=ax)
+    ax.set_ylabel('Mean |Δ log p(target)|')
+    ax.set_title('SAE interventions vs norm-matched random controls')
+    ax.tick_params(axis='x', rotation=0)
+    figure.tight_layout()
+    figure.savefig(fig_dir / 'causal_effects.png', dpi=160)
+    plt.close(figure)
+
+
+def main() -> None:
+    args = parse_args()
+    catalog = pd.read_csv(args.artifact_dir / 'feature_catalog.csv')
+    layers = pd.read_csv(args.artifact_dir / 'layer_metrics.csv')
+    stability = pd.read_csv(args.artifact_dir / 'stability.csv')
+    causal = pd.read_csv(args.artifact_dir / 'causal_results.csv')
+    selected = _selected_features(catalog)
+    _save_plots(args.artifact_dir, selected, layers, causal)
+
+    mean_auc = float(selected['auroc'].mean())
+    median_auc = float(selected['auroc'].median())
+    best_layer_row = layers.sort_values('linear_probe_macro_auroc', ascending=False).iloc[0]
+    mean_jaccard = float(stability['topk_jaccard'].mean())
+    mean_sparse_cos = float(stability['sparse_cosine'].mean())
+
+    sae = causal[causal['condition'] == 'sae_feature']
+    random = causal[causal['condition'] == 'random_norm_matched']
+    sae_abs = float(np.mean(np.abs(sae['target_logprob_delta'])))
+    random_abs = float(np.mean(np.abs(random['target_logprob_delta'])))
+    ratio = sae_abs / max(random_abs, 1e-12)
+    active_rate = float(np.mean(sae['feature_activation'] > 0))
+    top1_change = float(sae['top1_changed'].mean())
+
+    if mean_auc >= 0.8 and sae_abs < 0.08:
+        interpretation = (
+            'The selected sparse features were strongly predictive on held-out prompts, but their '
+            'interventions produced only modest downstream target-probability changes. This supports '
+            'the project’s central warning that representation-level correlation need not imply strong '
+            'causal control.'
+        )
+    elif mean_auc >= 0.8 and sae_abs >= 0.08 and ratio >= 1.5:
+        interpretation = (
+            'The selected sparse features were strongly predictive and their interventions produced '
+            'larger target-probability shifts than norm-matched random residual perturbations, providing '
+            'evidence that at least some predictive features also have downstream causal influence.'
+        )
+    elif mean_auc < 0.65:
+        interpretation = (
+            'Feature/concept predictiveness was limited on held-out prompts, so strong causal claims '
+            'would be premature. The main result is diagnostic: the chosen concepts or feature-selection '
+            'procedure should be refined before interpreting intervention effects.'
+        )
+    else:
+        interpretation = (
+            'The results show mixed predictive and causal evidence. FeatureLens therefore reports the '
+            'association and intervention measurements separately rather than collapsing them into a '
+            'single interpretability score.'
+        )
+
+    headline = (
+        f'Selected SAE features averaged {mean_auc:.3f} held-out AUROC; SAE interventions changed '
+        f'target log-probability by {sae_abs:.3f} on average versus {random_abs:.3f} for norm-matched '
+        'random residual controls.'
+    )
+    highlights = [
+        f'Median selected-feature held-out AUROC: {median_auc:.3f}.',
+        f'Best residual linear-probe layer: {int(best_layer_row["layer"])} with macro AUROC {best_layer_row["linear_probe_macro_auroc"]:.3f}.',
+        f'Mean paraphrase TopK Jaccard: {mean_jaccard:.3f}; sparse activation cosine: {mean_sparse_cos:.3f}.',
+        f'Selected feature active on {active_rate:.1%} of causal prompts; modified top-1 token on {top1_change:.1%}.',
+        f'Mean absolute causal effect / random-control effect ratio: {ratio:.2f}×.',
+    ]
+    summary = {
+        'headline': headline,
+        'highlights': highlights,
+        'interpretation': interpretation,
+        'metrics': {
+            'mean_selected_feature_test_auroc': mean_auc,
+            'median_selected_feature_test_auroc': median_auc,
+            'best_linear_probe_layer': int(best_layer_row['layer']),
+            'best_linear_probe_macro_auroc': float(best_layer_row['linear_probe_macro_auroc']),
+            'mean_paraphrase_topk_jaccard': mean_jaccard,
+            'mean_paraphrase_sparse_cosine': mean_sparse_cos,
+            'mean_abs_sae_target_logprob_delta': sae_abs,
+            'mean_abs_random_target_logprob_delta': random_abs,
+            'causal_to_random_effect_ratio': ratio,
+            'causal_prompt_feature_active_rate': active_rate,
+            'sae_top1_change_rate': top1_change,
+        },
+    }
+    (args.artifact_dir / 'summary.json').write_text(json.dumps(summary, indent=2), encoding='utf-8')
+
+    lines = [
+        '# FeatureLens experiment report',
+        '',
+        '## Research question',
+        '',
+        '**Do sparse features that predict a concept also causally influence model behaviour?**',
+        '',
+        '## Executive summary',
+        '',
+        headline,
+        '',
+        interpretation,
+        '',
+        '## Key measurements',
+        '',
+        *[f'- {item}' for item in highlights],
+        '',
+        '## Experimental design',
+        '',
+        '- Model: Qwen3-1.7B-Base.',
+        '- SAEs: Qwen-Scope residual-stream TopK SAEs at the configured early/middle/late layers.',
+        '- Discovery set: controlled concept prompts with paired paraphrases.',
+        '- Split discipline: paraphrase groups are kept entirely in train or held-out test.',
+        '- Feature selection: training-split AUROC and activation contrast; held-out AUROC/F1 are reported separately.',
+        '- Linear baseline: multinomial logistic regression on the dense residual stream.',
+        '- Causal edit: reconstruction-preserving decoder-direction delta patched into the original residual.',
+        '- Negative control: deterministic random residual direction matched to the SAE perturbation L2 norm.',
+        '- Behavioural metric: first-token target probability/log-probability, rank, JS divergence, and top-1 changes.',
+        '',
+        '## Figures',
+        '',
+        '![Feature AUROC](figures/feature_auroc.png)',
+        '',
+        '![Layer diagnostics](figures/layer_diagnostics.png)',
+        '',
+        '![Causal effects](figures/causal_effects.png)',
+        '',
+        '## Interpretation guardrails',
+        '',
+        'A high feature/concept AUROC is treated as correlational evidence only. Causal evidence requires a downstream change under intervention and is interpreted relative to the norm-matched random control. The narrative above is generated from saved metrics; no result values are hard-coded.',
+        '',
+        '## Reproducibility',
+        '',
+        'Run `python experiments/run_all.py` from the repository root. Raw activation matrices, splits, selected features, causal rows, figures, and this report are all materialized under `artifacts/`.',
+        '',
+    ]
+    (args.artifact_dir / 'report.md').write_text('\n'.join(lines), encoding='utf-8')
+    print(headline)
+    print(f'Wrote {args.artifact_dir / "report.md"}')
+
+
+if __name__ == '__main__':
+    main()

experiments/run_all.py

@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+import subprocess
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def run(script: str, *args: str) -> None:
+    module = f"experiments.{Path(script).stem}"
+    command = [sys.executable, '-m', module, *args]
+    print('\n$', ' '.join(command), flush=True)
+    subprocess.run(command, cwd=ROOT, check=True)
+
+
+def main() -> None:
+    run('build_dataset.py')
+    run('collect_activations.py')
+    run('evaluate_features.py')
+    run('run_causal.py')
+    run('make_report.py')
+    print('\nFeatureLens experiment pipeline complete. See artifacts/report.md')
+
+
+if __name__ == '__main__':
+    main()

experiments/run_causal.py

@@ -0,0 +1,195 @@
+from __future__ import annotations
+
+import argparse
+import csv
+import math
+from pathlib import Path
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from experiments.common import ARTIFACT_DIR, DATA_DIR, load_jsonl, set_seed
+from featurelens.config import SETTINGS
+from featurelens.interventions import InterventionSpec, normalized_random_control, residual_delta
+from featurelens.metrics import js_divergence_from_logits, safe_log_probability
+from featurelens.sae import SAEStore
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description='Run held-out causal SAE interventions.')
+    parser.add_argument('--tasks', type=Path, default=DATA_DIR / 'causal_tasks.jsonl')
+    parser.add_argument('--catalog', type=Path, default=ARTIFACT_DIR / 'feature_catalog.csv')
+    parser.add_argument('--output', type=Path, default=ARTIFACT_DIR / 'causal_results.csv')
+    parser.add_argument('--seed', type=int, default=42)
+    return parser.parse_args()
+
+
+def load_selected_features(path: Path) -> dict[str, dict]:
+    rows = []
+    with path.open(newline='', encoding='utf-8') as handle:
+        rows = list(csv.DictReader(handle))
+    selected: dict[str, dict] = {}
+    for row in rows:
+        concept = row['concept']
+        score = float(row['train_auroc'])
+        contrast = float(row['activation_rate_pos']) - float(row['activation_rate_neg'])
+        key = (score, contrast)
+        if concept not in selected or key > selected[concept]['_key']:
+            selected[concept] = {
+                '_key': key,
+                'layer': int(row['layer']),
+                'feature_id': int(row['feature_id']),
+                'train_auroc': score,
+                'test_auroc': float(row['auroc']),
+                'test_f1': float(row['f1']),
+            }
+    return selected
+
+
+def hidden_from_output(output):
+    return output[0] if isinstance(output, tuple) else output
+
+
+def replace_hidden(output, hidden):
+    return (hidden, *output[1:]) if isinstance(output, tuple) else hidden
+
+
+@torch.inference_mode()
+def main() -> None:
+    args = parse_args()
+    set_seed(args.seed)
+    tasks = load_jsonl(args.tasks)
+    selected = load_selected_features(args.catalog)
+    missing = sorted({task['concept'] for task in tasks}.difference(selected))
+    if missing:
+        raise RuntimeError(f'No selected SAE features for concepts: {missing}')
+
+    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+    model_dtype = torch.float16 if device.type == 'cuda' else torch.float32
+    tokenizer = AutoTokenizer.from_pretrained(SETTINGS.model_id)
+    if tokenizer.pad_token_id is None:
+        tokenizer.pad_token = tokenizer.eos_token
+    model = AutoModelForCausalLM.from_pretrained(
+        SETTINGS.model_id,
+        torch_dtype=model_dtype,
+        low_cpu_mem_usage=True,
+    ).to(device)
+    model.eval()
+    selected_layers = sorted({item['layer'] for item in selected.values()})
+    sae_store = SAEStore(
+        SETTINGS.sae_repo_id,
+        layers=selected_layers,
+        device=device,
+        dtype=torch.float32,
+        top_k=SETTINGS.sae_top_k,
+    )
+
+    results: list[dict] = []
+    for task_idx, task in enumerate(tasks):
+        concept = task['concept']
+        choice = selected[concept]
+        layer = int(choice['layer'])
+        feature_id = int(choice['feature_id'])
+        sae = sae_store.get(layer)
+        inputs = tokenizer(task['prompt'], return_tensors='pt', truncation=True, max_length=192)
+        inputs = {key: value.to(device) for key, value in inputs.items()}
+        capture: dict = {}
+
+        def capture_hook(_module, _inp, output):
+            if 'hidden' not in capture:
+                capture['hidden'] = hidden_from_output(output).detach()
+
+        handle = model.model.layers[layer].register_forward_hook(capture_hook)
+        baseline_out = model(**inputs, use_cache=False)
+        handle.remove()
+        baseline_logits = baseline_out.logits[0, -1]
+        residual = capture['hidden'][0, -1]
+        encoding = sae.encode(residual)
+        original_activation = encoding.activation_for(feature_id)
+
+        target_ids = tokenizer(task['target'], add_special_tokens=False)['input_ids']
+        if not target_ids:
+            raise RuntimeError(f"Target tokenization empty for task {task['id']}")
+        target_id = int(target_ids[0])
+        baseline_prob = float(torch.softmax(baseline_logits.float(), dim=-1)[target_id].item())
+        baseline_rank = int((baseline_logits > baseline_logits[target_id]).sum().item()) + 1
+        baseline_top1 = int(torch.argmax(baseline_logits).item())
+
+        specs = [
+            ('ablate', InterventionSpec('ablate', 0.0)),
+            ('amplify_2x', InterventionSpec('scale', 2.0)),
+        ]
+        for intervention_name, spec in specs:
+            delta = residual_delta(sae.decoder_direction(feature_id), original_activation, spec)
+            control_delta = normalized_random_control(delta, seed=args.seed + task_idx * 17 + len(results))
+
+            for condition, applied_delta in (
+                ('sae_feature', delta),
+                ('random_norm_matched', control_delta),
+            ):
+                applied = {'done': False}
+
+                def edit_hook(_module, _inp, output, *, edit=applied_delta, state=applied):
+                    if state['done']:
+                        return output
+                    hidden = hidden_from_output(output)
+                    modified = hidden.clone()
+                    modified[:, -1, :] = modified[:, -1, :] + edit.to(hidden.device, hidden.dtype)
+                    state['done'] = True
+                    return replace_hidden(output, modified)
+
+                hook = model.model.layers[layer].register_forward_hook(edit_hook)
+                modified_out = model(**inputs, use_cache=False)
+                hook.remove()
+                modified_logits = modified_out.logits[0, -1]
+                modified_prob = float(
+                    torch.softmax(modified_logits.float(), dim=-1)[target_id].item()
+                )
+                modified_rank = int((modified_logits > modified_logits[target_id]).sum().item()) + 1
+                modified_top1 = int(torch.argmax(modified_logits).item())
+                results.append(
+                    {
+                        'task_id': task['id'],
+                        'concept': concept,
+                        'prompt': task['prompt'],
+                        'target_text': task['target'],
+                        'target_first_token': tokenizer.decode([target_id]),
+                        'target_token_count': len(target_ids),
+                        'layer': layer,
+                        'feature_id': feature_id,
+                        'feature_train_auroc': choice['train_auroc'],
+                        'feature_test_auroc': choice['test_auroc'],
+                        'feature_test_f1': choice['test_f1'],
+                        'feature_activation': original_activation,
+                        'intervention': intervention_name,
+                        'condition': condition,
+                        'delta_activation': spec.delta_activation(original_activation)
+                        if condition == 'sae_feature'
+                        else math.nan,
+                        'perturbation_l2': float(torch.linalg.vector_norm(applied_delta.float()).item()),
+                        'baseline_target_prob': baseline_prob,
+                        'modified_target_prob': modified_prob,
+                        'target_prob_delta': modified_prob - baseline_prob,
+                        'target_logprob_delta': safe_log_probability(modified_prob)
+                        - safe_log_probability(baseline_prob),
+                        'baseline_target_rank': baseline_rank,
+                        'modified_target_rank': modified_rank,
+                        'target_rank_delta': modified_rank - baseline_rank,
+                        'js_divergence': js_divergence_from_logits(
+                            baseline_logits, modified_logits
+                        ),
+                        'top1_changed': int(modified_top1 != baseline_top1),
+                    }
+                )
+        print(f"Causal task {task_idx + 1}/{len(tasks)}: {concept}", flush=True)
+
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    with args.output.open('w', newline='', encoding='utf-8') as handle:
+        writer = csv.DictWriter(handle, fieldnames=list(results[0].keys()))
+        writer.writeheader()
+        writer.writerows(results)
+    print(f'Wrote {len(results)} causal intervention rows to {args.output}')
+
+
+if __name__ == '__main__':
+    main()

experiments/split.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+import random
+
+
+def grouped_concept_split(rows: list[dict], test_fraction: float = 0.25, seed: int = 42):
+    """Split paraphrase groups within each concept so paired prompts never leak across splits."""
+    rng = random.Random(seed)
+    train_ids: list[int] = []
+    test_ids: list[int] = []
+    concepts = sorted({row['concept'] for row in rows})
+    for concept in concepts:
+        concept_rows = [(idx, row) for idx, row in enumerate(rows) if row['concept'] == concept]
+        pair_ids = sorted({row['pair_id'] for _, row in concept_rows})
+        rng.shuffle(pair_ids)
+        n_test = max(1, round(len(pair_ids) * test_fraction))
+        test_pairs = set(pair_ids[:n_test])
+        for idx, row in concept_rows:
+            (test_ids if row['pair_id'] in test_pairs else train_ids).append(idx)
+    return sorted(train_ids), sorted(test_ids)

featurelens/__init__.py

@@ -0,0 +1,14 @@
+"""FeatureLens: causal sparse-feature interpretability for Qwen3."""
+
+from .config import SETTINGS, Settings
+from .interventions import InterventionSpec
+from .sae import SAEStore, SAEWeights, SparseEncoding
+
+__all__ = [
+    'SETTINGS',
+    'Settings',
+    'InterventionSpec',
+    'SAEStore',
+    'SAEWeights',
+    'SparseEncoding',
+]

featurelens/catalog.py

@@ -0,0 +1,47 @@
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+
+
+class FeatureCatalog:
+    def __init__(self, artifact_dir: str | Path = 'artifacts') -> None:
+        self.artifact_dir = Path(artifact_dir)
+        self.rows: list[dict] = []
+        path = self.artifact_dir / 'feature_catalog.csv'
+        if path.exists():
+            with path.open(newline='', encoding='utf-8') as handle:
+                self.rows = list(csv.DictReader(handle))
+
+    def hint(self, layer: int, feature_id: int) -> str:
+        matches = [
+            row
+            for row in self.rows
+            if int(row.get('layer', -1)) == int(layer)
+            and int(row.get('feature_id', -1)) == int(feature_id)
+        ]
+        if not matches:
+            return 'unlabeled'
+        best = max(
+            matches,
+            key=lambda row: float(row.get('train_auroc', row.get('auroc', 0.0)) or 0.0),
+        )
+        concept = best.get('concept', 'unlabeled')
+        auc = float(best.get('auroc', 0.0) or 0.0)
+        return f'{concept} (AUROC {auc:.2f})'
+
+    def benchmark_markdown(self) -> str:
+        summary_path = self.artifact_dir / 'summary.json'
+        if not summary_path.exists():
+            return (
+                '### Offline benchmark\n\n'
+                'No benchmark artifacts are committed yet. Run `python experiments/run_all.py` '
+                'on a CUDA machine, then commit `artifacts/summary.json`, `feature_catalog.csv`, '
+                'and `report.md`. The live workbench is fully usable without them.'
+            )
+        data = json.loads(summary_path.read_text(encoding='utf-8'))
+        headline = data.get('headline', 'Benchmark completed.')
+        bullets = data.get('highlights', [])
+        body = '\n'.join(f'- {item}' for item in bullets)
+        return f'### Offline benchmark\n\n{headline}\n\n{body}'

featurelens/config.py

@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+
+def _parse_layers(raw: str | None) -> tuple[int, ...]:
+    if not raw:
+        return (4, 14, 26)
+    layers = tuple(sorted({int(x.strip()) for x in raw.split(',') if x.strip()}))
+    if not layers:
+        raise ValueError('FEATURELENS_LAYERS must contain at least one layer.')
+    if any(layer < 0 or layer > 27 for layer in layers):
+        raise ValueError('Qwen3-1.7B has residual-stream SAE layers 0-27.')
+    return layers
+
+
+@dataclass(frozen=True)
+class Settings:
+    model_id: str = os.getenv('FEATURELENS_MODEL_ID', 'Qwen/Qwen3-1.7B-Base')
+    sae_repo_id: str = os.getenv(
+        'FEATURELENS_SAE_REPO', 'Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50'
+    )
+    layers: tuple[int, ...] = _parse_layers(os.getenv('FEATURELENS_LAYERS'))
+    sae_top_k: int = int(os.getenv('FEATURELENS_SAE_TOP_K', '50'))
+    sae_width: int = 32_768
+    d_model: int = 2_048
+    max_prompt_tokens: int = int(os.getenv('FEATURELENS_MAX_PROMPT_TOKENS', '256'))
+    max_new_tokens: int = int(os.getenv('FEATURELENS_MAX_NEW_TOKENS', '32'))
+    eager_load: bool = os.getenv(
+        'FEATURELENS_EAGER_LOAD', '1' if os.getenv('SPACE_ID') else '0'
+    ).lower() in {'1', 'true', 'yes', 'on'}
+    sae_dtype: str = os.getenv(
+        'FEATURELENS_SAE_DTYPE', 'float16' if os.getenv('SPACE_ID') else 'float32'
+    )
+
+
+SETTINGS = Settings()

featurelens/hf_runtime.py

@@ -0,0 +1,16 @@
+from __future__ import annotations
+
+try:
+    import spaces  # type: ignore
+except ImportError:  # Local development and unit tests.
+    spaces = None
+
+
+def gpu(duration: int = 60):
+    if spaces is not None and hasattr(spaces, "GPU"):
+        return spaces.GPU(duration=duration)
+
+    def decorator(fn):
+        return fn
+
+    return decorator

featurelens/interventions.py

@@ -0,0 +1,44 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+
+
+@dataclass(frozen=True)
+class InterventionSpec:
+    mode: str
+    coefficient: float
+
+    def delta_activation(self, original_activation: float) -> float:
+        mode = self.mode.lower().strip()
+        original = float(original_activation)
+        coefficient = float(self.coefficient)
+        if mode == 'ablate':
+            return -original
+        if mode == 'scale':
+            return (coefficient - 1.0) * original
+        if mode == 'inject':
+            return coefficient
+        raise ValueError("mode must be one of: 'ablate', 'scale', 'inject'.")
+
+
+def residual_delta(
+    decoder_direction: torch.Tensor,
+    original_activation: float,
+    spec: InterventionSpec,
+) -> torch.Tensor:
+    """Return the reconstruction-preserving SAE delta applied to the original residual."""
+    return decoder_direction * spec.delta_activation(original_activation)
+
+
+def normalized_random_control(delta: torch.Tensor, seed: int) -> torch.Tensor:
+    """Generate a deterministic random residual perturbation with identical L2 norm."""
+    norm = torch.linalg.vector_norm(delta.float())
+    if float(norm.item()) == 0.0:
+        return torch.zeros_like(delta)
+    generator = torch.Generator(device='cpu').manual_seed(int(seed))
+    random_vec = torch.randn(delta.shape, generator=generator, dtype=torch.float32)
+    random_vec = random_vec / torch.linalg.vector_norm(random_vec)
+    random_vec = random_vec * norm.cpu()
+    return random_vec.to(device=delta.device, dtype=delta.dtype)

featurelens/metrics.py

@@ -0,0 +1,43 @@
+from __future__ import annotations
+
+import math
+
+import numpy as np
+import torch
+
+
+def reconstruction_metrics(original: torch.Tensor, reconstructed: torch.Tensor) -> dict[str, float]:
+    x = original.float().reshape(-1)
+    x_hat = reconstructed.float().reshape(-1)
+    mse = torch.mean((x - x_hat) ** 2)
+    denom = torch.mean(x**2).clamp_min(1e-12)
+    nmse = mse / denom
+    cosine = torch.nn.functional.cosine_similarity(x.unsqueeze(0), x_hat.unsqueeze(0)).item()
+    return {
+        'mse': float(mse.item()),
+        'nmse': float(nmse.item()),
+        'cosine': float(cosine),
+    }
+
+
+def js_divergence_from_logits(logits_a: torch.Tensor, logits_b: torch.Tensor) -> float:
+    p = torch.softmax(logits_a.float(), dim=-1)
+    q = torch.softmax(logits_b.float(), dim=-1)
+    m = 0.5 * (p + q)
+    eps = 1e-12
+    kl_pm = torch.sum(p * (torch.log(p + eps) - torch.log(m + eps)))
+    kl_qm = torch.sum(q * (torch.log(q + eps) - torch.log(m + eps)))
+    return float((0.5 * (kl_pm + kl_qm)).item())
+
+
+def safe_log_probability(probability: float) -> float:
+    return math.log(max(float(probability), 1e-12))
+
+
+def sparse_jaccard(indices_a: np.ndarray, values_a: np.ndarray, indices_b: np.ndarray, values_b: np.ndarray) -> float:
+    a = set(indices_a[np.asarray(values_a) > 0].tolist())
+    b = set(indices_b[np.asarray(values_b) > 0].tolist())
+    union = a | b
+    if not union:
+        return 1.0
+    return len(a & b) / len(union)

featurelens/runtime.py

@@ -0,0 +1,336 @@
+from __future__ import annotations
+
+import html
+import os
+from collections.abc import Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from .catalog import FeatureCatalog
+from .config import SETTINGS, Settings
+from .interventions import InterventionSpec, residual_delta
+from .metrics import js_divergence_from_logits, reconstruction_metrics, safe_log_probability
+from .sae import SAEStore, SparseEncoding
+
+
+def _dtype_from_name(name: str) -> torch.dtype:
+    name = name.lower().strip()
+    if name in {'float16', 'fp16', 'half'}:
+        return torch.float16
+    if name in {'bfloat16', 'bf16'}:
+        return torch.bfloat16
+    if name in {'float32', 'fp32'}:
+        return torch.float32
+    raise ValueError(f'Unsupported dtype: {name}')
+
+
+def _default_device() -> torch.device:
+    # ZeroGPU exposes CUDA emulation at module load time. Force the recommended
+    # CUDA placement on Spaces even if a local availability probe is conservative.
+    if os.getenv('SPACE_ID'):
+        return torch.device('cuda')
+    if torch.cuda.is_available():
+        return torch.device('cuda')
+    return torch.device('cpu')
+
+
+@dataclass
+class AnalysisResult:
+    tokens: list[str]
+    token_index: int
+    layer: int
+    features: SparseEncoding
+    rows: list[list[object]]
+    metrics: dict[str, float]
+
+
+@dataclass
+class InterventionResult:
+    baseline_text: str
+    modified_text: str
+    feature_activation: float
+    delta_activation: float
+    perturbation_norm: float
+    js_divergence: float
+    target_text: str
+    target_token: str
+    target_token_count: int
+    baseline_target_prob: float | None
+    modified_target_prob: float | None
+    target_logprob_delta: float | None
+    top_token_rows: list[list[object]]
+
+
+class FeatureLensRuntime:
+    def __init__(self, settings: Settings = SETTINGS) -> None:
+        self.settings = settings
+        self.device = _default_device()
+        self.model_dtype = torch.float16 if self.device.type == 'cuda' else torch.float32
+        self.sae_dtype = _dtype_from_name(settings.sae_dtype)
+        if self.device.type == 'cpu' and self.sae_dtype != torch.float32:
+            self.sae_dtype = torch.float32
+        self.model = None
+        self.tokenizer = None
+        self.sae_store: SAEStore | None = None
+        self.catalog = FeatureCatalog()
+        self.load_error: str | None = None
+
+    def ensure_ready(self, preload_saes: bool = False) -> None:
+        if self.model is not None and self.tokenizer is not None and self.sae_store is not None:
+            if preload_saes:
+                self.sae_store.preload()
+            return
+        self.tokenizer = AutoTokenizer.from_pretrained(self.settings.model_id)
+        if self.tokenizer.pad_token_id is None:
+            self.tokenizer.pad_token = self.tokenizer.eos_token
+        self.tokenizer.padding_side = 'left'
+        self.model = AutoModelForCausalLM.from_pretrained(
+            self.settings.model_id,
+            torch_dtype=self.model_dtype,
+            low_cpu_mem_usage=True,
+        )
+        self.model.to(self.device)
+        self.model.eval()
+        self.sae_store = SAEStore(
+            repo_id=self.settings.sae_repo_id,
+            layers=self.settings.layers,
+            device=self.device,
+            dtype=self.sae_dtype,
+            top_k=self.settings.sae_top_k,
+        )
+        if preload_saes:
+            self.sae_store.preload()
+
+    def token_choices(self, text: str) -> list[tuple[str, int]]:
+        self.ensure_ready(preload_saes=False)
+        assert self.tokenizer is not None
+        ids = self.tokenizer(text, add_special_tokens=True)['input_ids']
+        tokens = [self.tokenizer.decode([token_id]) for token_id in ids]
+        return [(f'{idx}: {token!r}', idx) for idx, token in enumerate(tokens)]
+
+    def _inputs(self, text: str) -> dict[str, torch.Tensor]:
+        assert self.tokenizer is not None
+        batch = self.tokenizer(
+            text,
+            return_tensors='pt',
+            truncation=True,
+            max_length=self.settings.max_prompt_tokens,
+        )
+        return {key: value.to(self.device) for key, value in batch.items()}
+
+    @staticmethod
+    def _hidden_from_output(output):
+        return output[0] if isinstance(output, tuple) else output
+
+    @staticmethod
+    def _replace_hidden_in_output(output, hidden: torch.Tensor):
+        if isinstance(output, tuple):
+            return (hidden, *output[1:])
+        return hidden
+
+    @contextmanager
+    def _capture_hook(self, layer: int, bucket: dict) -> Iterator[None]:
+        assert self.model is not None
+
+        def hook(_module, _inputs, output):
+            hidden = self._hidden_from_output(output)
+            if 'hidden' not in bucket:
+                bucket['hidden'] = hidden.detach()
+
+        handle = self.model.model.layers[int(layer)].register_forward_hook(hook)
+        try:
+            yield
+        finally:
+            handle.remove()
+
+    @contextmanager
+    def _delta_hook(self, layer: int, token_index: int, delta: torch.Tensor) -> Iterator[None]:
+        assert self.model is not None
+        applied = {'done': False}
+
+        def hook(_module, _inputs, output):
+            if applied['done']:
+                return output
+            hidden = self._hidden_from_output(output)
+            if hidden.ndim != 3:
+                return output
+            seq_len = hidden.shape[1]
+            idx = int(token_index)
+            if idx < 0:
+                idx = seq_len + idx
+            if idx < 0 or idx >= seq_len:
+                raise IndexError(f'Token index {token_index} outside prompt length {seq_len}.')
+            modified = hidden.clone()
+            modified[:, idx, :] = modified[:, idx, :] + delta.to(hidden.device, hidden.dtype)
+            applied['done'] = True
+            return self._replace_hidden_in_output(output, modified)
+
+        handle = self.model.model.layers[int(layer)].register_forward_hook(hook)
+        try:
+            yield
+        finally:
+            handle.remove()
+
+    @torch.inference_mode()
+    def analyze(self, text: str, layer: int, token_index: int = -1, top_n: int = 12) -> AnalysisResult:
+        self.ensure_ready(preload_saes=False)
+        assert self.model is not None and self.tokenizer is not None and self.sae_store is not None
+        if int(layer) not in self.settings.layers:
+            raise ValueError(f'Layer must be one of {self.settings.layers}.')
+        inputs = self._inputs(text)
+        bucket: dict = {}
+        with self._capture_hook(int(layer), bucket):
+            self.model(**inputs, use_cache=False)
+        hidden = bucket['hidden'][0]
+        seq_len = hidden.shape[0]
+        idx = int(token_index)
+        if idx < 0:
+            idx = seq_len + idx
+        if idx < 0 or idx >= seq_len:
+            raise IndexError(f'Token index {token_index} outside prompt length {seq_len}.')
+        residual = hidden[idx]
+        sae = self.sae_store.get(int(layer))
+        encoding = sae.encode(residual)
+        reconstruction = sae.decode_sparse(encoding)
+        metrics = reconstruction_metrics(residual, reconstruction)
+        metrics['active_features'] = float(encoding.active_count)
+        ids = inputs['input_ids'][0].tolist()
+        tokens = [self.tokenizer.decode([token_id]) for token_id in ids]
+        rows: list[list[object]] = []
+        rank_count = min(int(top_n), encoding.indices.numel())
+        for rank in range(rank_count):
+            feature_id = int(encoding.indices[rank].item())
+            activation = float(encoding.values[rank].item())
+            rows.append(
+                [rank + 1, feature_id, activation, self.catalog.hint(int(layer), feature_id)]
+            )
+        return AnalysisResult(
+            tokens=tokens,
+            token_index=idx,
+            layer=int(layer),
+            features=encoding,
+            rows=rows,
+            metrics=metrics,
+        )
+
+    def token_html(self, tokens: list[str], selected_index: int) -> str:
+        chips = []
+        for idx, token in enumerate(tokens):
+            safe = html.escape(token if token.strip() else repr(token))
+            selected = idx == int(selected_index)
+            cls = 'token selected' if selected else 'token'
+            chips.append(f'<span class="{cls}"><sup>{idx}</sup>{safe}</span>')
+        return '<div class="token-wrap">' + ''.join(chips) + '</div>'
+
+    @staticmethod
+    def _top_token_rows(tokenizer, baseline_logits: torch.Tensor, modified_logits: torch.Tensor, k: int = 8):
+        p = torch.softmax(baseline_logits.float(), dim=-1)
+        q = torch.softmax(modified_logits.float(), dim=-1)
+        union_ids = torch.unique(torch.cat([torch.topk(p, k).indices, torch.topk(q, k).indices]))
+        rows = []
+        for token_id in union_ids.tolist():
+            token = tokenizer.decode([int(token_id)])
+            bp = float(p[token_id].item())
+            mp = float(q[token_id].item())
+            rows.append([repr(token), bp, mp, mp - bp])
+        rows.sort(key=lambda row: max(row[1], row[2]), reverse=True)
+        return rows[: min(len(rows), 12)]
+
+    @torch.inference_mode()
+    def intervene(
+        self,
+        text: str,
+        layer: int,
+        token_index: int,
+        feature_id: int,
+        mode: str,
+        coefficient: float,
+        target_text: str = '',
+        max_new_tokens: int = 24,
+    ) -> InterventionResult:
+        self.ensure_ready(preload_saes=False)
+        assert self.model is not None and self.tokenizer is not None and self.sae_store is not None
+        inputs = self._inputs(text)
+        prompt_len = int(inputs['input_ids'].shape[1])
+        idx = int(token_index)
+        if idx < 0:
+            idx = prompt_len + idx
+        if idx < 0 or idx >= prompt_len:
+            raise IndexError(f'Token index {token_index} outside prompt length {prompt_len}.')
+
+        sae = self.sae_store.get(int(layer))
+        capture: dict = {}
+        generation_kwargs = {
+            'max_new_tokens': min(int(max_new_tokens), self.settings.max_new_tokens),
+            'do_sample': False,
+            'return_dict_in_generate': True,
+            'output_scores': True,
+            'pad_token_id': self.tokenizer.eos_token_id,
+        }
+        with self._capture_hook(int(layer), capture):
+            baseline = self.model.generate(**inputs, **generation_kwargs)
+        residual = capture['hidden'][0, idx]
+        encoding = sae.encode(residual)
+        original_activation = encoding.activation_for(int(feature_id))
+        spec = InterventionSpec(mode=mode, coefficient=float(coefficient))
+        delta = residual_delta(sae.decoder_direction(int(feature_id)), original_activation, spec)
+
+        with self._delta_hook(int(layer), idx, delta):
+            modified = self.model.generate(**inputs, **generation_kwargs)
+
+        baseline_ids = baseline.sequences[0, prompt_len:]
+        modified_ids = modified.sequences[0, prompt_len:]
+        baseline_text = self.tokenizer.decode(baseline_ids, skip_special_tokens=True)
+        modified_text = self.tokenizer.decode(modified_ids, skip_special_tokens=True)
+
+        if not baseline.scores or not modified.scores:
+            raise RuntimeError('Generation returned no score tensors.')
+        baseline_logits = baseline.scores[0][0]
+        modified_logits = modified.scores[0][0]
+        js = js_divergence_from_logits(baseline_logits, modified_logits)
+
+        target_token = ''
+        target_count = 0
+        bp = mp = log_delta = None
+        if target_text.strip():
+            target_ids = self.tokenizer(target_text, add_special_tokens=False)['input_ids']
+            target_count = len(target_ids)
+            if target_ids:
+                target_id = int(target_ids[0])
+                target_token = self.tokenizer.decode([target_id])
+                p = torch.softmax(baseline_logits.float(), dim=-1)
+                q = torch.softmax(modified_logits.float(), dim=-1)
+                bp = float(p[target_id].item())
+                mp = float(q[target_id].item())
+                log_delta = safe_log_probability(mp) - safe_log_probability(bp)
+
+        return InterventionResult(
+            baseline_text=baseline_text,
+            modified_text=modified_text,
+            feature_activation=float(original_activation),
+            delta_activation=float(spec.delta_activation(original_activation)),
+            perturbation_norm=float(torch.linalg.vector_norm(delta.float()).item()),
+            js_divergence=float(js),
+            target_text=target_text,
+            target_token=target_token,
+            target_token_count=target_count,
+            baseline_target_prob=bp,
+            modified_target_prob=mp,
+            target_logprob_delta=log_delta,
+            top_token_rows=self._top_token_rows(
+                self.tokenizer, baseline_logits, modified_logits, k=8
+            ),
+        )
+
+
+RUNTIME = FeatureLensRuntime()
+
+if SETTINGS.eager_load:
+    try:
+        RUNTIME.ensure_ready(preload_saes=True)
+    except Exception as exc:  # Keep the UI alive and surface a useful error on first call.
+        RUNTIME.load_error = f'{type(exc).__name__}: {exc}'
+        print(f'FeatureLens eager load failed: {RUNTIME.load_error}')

featurelens/sae.py

@@ -0,0 +1,125 @@
+from __future__ import annotations
+
+from collections.abc import Iterable
+from dataclasses import dataclass
+from pathlib import Path
+
+import torch
+from huggingface_hub import hf_hub_download
+
+
+@dataclass
+class SparseEncoding:
+    indices: torch.Tensor
+    values: torch.Tensor
+    pre_activations: torch.Tensor | None = None
+
+    @property
+    def active_count(self) -> int:
+        return int((self.values > 0).sum().item())
+
+    def activation_for(self, feature_id: int) -> float:
+        mask = self.indices == int(feature_id)
+        if not bool(mask.any()):
+            return 0.0
+        return float(self.values[mask][0].item())
+
+
+@dataclass
+class SAEWeights:
+    layer: int
+    w_enc_t: torch.Tensor  # [d_model, d_sae]
+    w_dec: torch.Tensor  # [d_model, d_sae]
+    b_enc: torch.Tensor  # [d_sae]
+    b_dec: torch.Tensor  # [d_model]
+    top_k: int = 50
+
+    @torch.inference_mode()
+    def encode(self, hidden: torch.Tensor, return_pre: bool = False) -> SparseEncoding:
+        """Encode one or more residual vectors without materializing dense sparse acts."""
+        if hidden.shape[-1] != self.w_enc_t.shape[0]:
+            raise ValueError(
+                f'Expected hidden dim {self.w_enc_t.shape[0]}, got {hidden.shape[-1]}.'
+            )
+        compute_hidden = hidden.to(device=self.w_enc_t.device, dtype=self.w_enc_t.dtype)
+        pre = compute_hidden @ self.w_enc_t + self.b_enc
+        relu = torch.relu(pre)
+        values, indices = torch.topk(relu, k=self.top_k, dim=-1)
+        return SparseEncoding(indices=indices, values=values, pre_activations=pre if return_pre else None)
+
+    @torch.inference_mode()
+    def decode_sparse(self, encoding: SparseEncoding) -> torch.Tensor:
+        """Decode TopK features efficiently using only selected decoder columns."""
+        indices = encoding.indices
+        values = encoding.values.to(device=self.w_dec.device, dtype=self.w_dec.dtype)
+        if indices.ndim == 1:
+            cols = self.w_dec[:, indices]  # [d_model, k]
+            return self.b_dec + cols @ values
+        flat_idx = indices.reshape(-1, indices.shape[-1])
+        flat_vals = values.reshape(-1, values.shape[-1])
+        outputs = []
+        for row_idx, row_vals in zip(flat_idx, flat_vals, strict=True):
+            cols = self.w_dec[:, row_idx]
+            outputs.append(self.b_dec + cols @ row_vals)
+        return torch.stack(outputs).reshape(*indices.shape[:-1], self.w_dec.shape[0])
+
+    def decoder_direction(self, feature_id: int) -> torch.Tensor:
+        if feature_id < 0 or feature_id >= self.w_dec.shape[1]:
+            raise ValueError(f'Feature id must be in [0, {self.w_dec.shape[1] - 1}].')
+        return self.w_dec[:, int(feature_id)]
+
+
+class SAEStore:
+    def __init__(
+        self,
+        repo_id: str,
+        layers: Iterable[int],
+        device: torch.device,
+        dtype: torch.dtype,
+        top_k: int = 50,
+        cache_dir: str | Path | None = None,
+    ) -> None:
+        self.repo_id = repo_id
+        self.layers = tuple(int(x) for x in layers)
+        self.device = device
+        self.dtype = dtype
+        self.top_k = int(top_k)
+        self.cache_dir = str(cache_dir) if cache_dir else None
+        self._cache: dict[int, SAEWeights] = {}
+
+    def get(self, layer: int) -> SAEWeights:
+        layer = int(layer)
+        if layer not in self.layers:
+            raise ValueError(f'Layer {layer} is not configured. Available: {self.layers}.')
+        if layer in self._cache:
+            return self._cache[layer]
+
+        path = hf_hub_download(
+            repo_id=self.repo_id,
+            filename=f'layer{layer}.sae.pt',
+            cache_dir=self.cache_dir,
+        )
+        try:
+            raw = torch.load(path, map_location='cpu', weights_only=True)
+        except TypeError:  # pragma: no cover - old torch fallback
+            raw = torch.load(path, map_location='cpu')
+
+        required = {'W_enc', 'W_dec', 'b_enc', 'b_dec'}
+        missing = required.difference(raw)
+        if missing:
+            raise KeyError(f'SAE checkpoint layer {layer} missing keys: {sorted(missing)}')
+
+        sae = SAEWeights(
+            layer=layer,
+            w_enc_t=raw['W_enc'].T.contiguous().to(self.device, dtype=self.dtype),
+            w_dec=raw['W_dec'].contiguous().to(self.device, dtype=self.dtype),
+            b_enc=raw['b_enc'].contiguous().to(self.device, dtype=self.dtype),
+            b_dec=raw['b_dec'].contiguous().to(self.device, dtype=self.dtype),
+            top_k=self.top_k,
+        )
+        self._cache[layer] = sae
+        return sae
+
+    def preload(self) -> None:
+        for layer in self.layers:
+            self.get(layer)

pyproject.toml

@@ -0,0 +1,18 @@
+[project]
+name = "featurelens"
+version = "0.1.0"
+description = "Causal sparse-feature interpretability workbench for Qwen3 and Qwen-Scope SAEs"
+requires-python = ">=3.10"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+exclude = ["artifacts"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "B", "UP"]
+ignore = ["E501"]

requirements-dev.txt

@@ -0,0 +1,3 @@
+-r requirements.txt
+pytest>=8.3,<10
+ruff>=0.9,<1

requirements.txt

@@ -0,0 +1,9 @@
+torch>=2.8.0,<2.12.0
+transformers>=4.51.0,<6.0.0
+huggingface_hub>=0.34.0,<2.0.0
+gradio>=6.0.0,<7.0.0
+numpy>=2.0.0,<3.0.0
+scipy>=1.14.0,<2.0.0
+scikit-learn>=1.6.0,<2.0.0
+pandas>=2.2.0,<3.0.0
+matplotlib>=3.9.0,<4.0.0

research_config.json

@@ -0,0 +1,26 @@
+{
+  "research_question": "Do sparse features that predict a concept also causally influence model behaviour?",
+  "model_id": "Qwen/Qwen3-1.7B-Base",
+  "sae_repo_id": "Qwen/SAE-Res-Qwen3-1.7B-Base-W32K-L0_50",
+  "layers": [4, 14, 26],
+  "sae_top_k": 50,
+  "sae_width": 32768,
+  "concepts": [
+    "code",
+    "mathematics",
+    "positive_sentiment",
+    "negative_sentiment",
+    "french_language",
+    "factual_entities",
+    "uncertainty"
+  ],
+  "discovery_prompts": 224,
+  "paraphrase_pairs_per_concept": 16,
+  "causal_tasks": 28,
+  "split_seed": 42,
+  "feature_selection": "training-split AUROC with activation-rate contrast tie-break",
+  "held_out_metrics": ["AUROC", "F1"],
+  "causal_interventions": ["ablate", "scale_2x"],
+  "negative_control": "norm-matched random residual direction",
+  "primary_causal_metric": "target first-token log-probability delta"
+}

scripts/release_check.py

@@ -0,0 +1,64 @@
+from __future__ import annotations
+
+import json
+from collections import Counter
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+REQUIRED = [
+    'README.md',
+    'app.py',
+    'requirements.txt',
+    'research_config.json',
+    'featurelens/runtime.py',
+    'featurelens/sae.py',
+    'featurelens/interventions.py',
+    'experiments/run_all.py',
+    'data/prompts.jsonl',
+    'data/causal_tasks.jsonl',
+]
+
+
+def load_jsonl(path: Path) -> list[dict]:
+    return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line]
+
+
+def main() -> None:
+    missing = [path for path in REQUIRED if not (ROOT / path).exists()]
+    if missing:
+        raise SystemExit(f'Missing required files: {missing}')
+
+    config = json.loads((ROOT / 'research_config.json').read_text(encoding='utf-8'))
+    assert config['layers'] == [4, 14, 26]
+    assert config['model_id'] == 'Qwen/Qwen3-1.7B-Base'
+    assert config['sae_width'] == 32768
+
+    prompts = load_jsonl(ROOT / 'data' / 'prompts.jsonl')
+    causal = load_jsonl(ROOT / 'data' / 'causal_tasks.jsonl')
+    assert len(prompts) == config['discovery_prompts']
+    assert len(causal) == config['causal_tasks']
+    concept_counts = Counter(row['concept'] for row in prompts)
+    assert len(concept_counts) == len(config['concepts'])
+    assert len(set(concept_counts.values())) == 1
+
+    oversized = []
+    for path in ROOT.rglob('*'):
+        if path.is_file() and '.git' not in path.parts and path.stat().st_size > 5_000_000:
+            oversized.append(str(path.relative_to(ROOT)))
+    if oversized:
+        raise SystemExit(f'Repository contains unexpectedly large tracked candidates: {oversized}')
+
+    readme = (ROOT / 'README.md').read_text(encoding='utf-8')
+    assert 'sdk: gradio' in readme
+    assert 'Qwen/Qwen3-1.7B-Base' in readme
+    assert 'norm-matched random' in readme.lower()
+
+    print('FeatureLens release check: PASS')
+    print(f'  discovery prompts: {len(prompts)}')
+    print(f'  causal tasks: {len(causal)}')
+    print(f'  layers: {config["layers"]}')
+
+
+if __name__ == '__main__':
+    main()

tests/test_catalog.py

@@ -0,0 +1,15 @@
+from __future__ import annotations
+
+from pathlib import Path
+
+from featurelens.catalog import FeatureCatalog
+
+
+def test_catalog_returns_best_hint(tmp_path: Path) -> None:
+    (tmp_path / 'feature_catalog.csv').write_text(
+        'layer,concept,feature_id,auroc\n14,math,123,0.80\n14,code,123,0.65\n',
+        encoding='utf-8',
+    )
+    catalog = FeatureCatalog(tmp_path)
+    assert catalog.hint(14, 123) == 'math (AUROC 0.80)'
+    assert catalog.hint(4, 123) == 'unlabeled'

tests/test_data.py

@@ -0,0 +1,27 @@
+from __future__ import annotations
+
+import json
+from collections import Counter
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def read_jsonl(path: Path):
+    return [json.loads(line) for line in path.read_text(encoding='utf-8').splitlines() if line]
+
+
+def test_discovery_dataset_is_balanced_and_paired() -> None:
+    rows = read_jsonl(ROOT / 'data' / 'prompts.jsonl')
+    assert len(rows) == 224
+    concept_counts = Counter(row['concept'] for row in rows)
+    assert set(concept_counts.values()) == {32}
+    pair_counts = Counter(row['pair_id'] for row in rows)
+    assert set(pair_counts.values()) == {2}
+
+
+def test_causal_dataset_covers_every_discovery_concept() -> None:
+    discovery = read_jsonl(ROOT / 'data' / 'prompts.jsonl')
+    causal = read_jsonl(ROOT / 'data' / 'causal_tasks.jsonl')
+    assert len(causal) == 28
+    assert {row['concept'] for row in causal} == {row['concept'] for row in discovery}

tests/test_interventions.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+import torch
+
+from featurelens.interventions import InterventionSpec, normalized_random_control, residual_delta
+
+
+def test_ablation_delta() -> None:
+    spec = InterventionSpec('ablate', 0.0)
+    assert spec.delta_activation(3.5) == -3.5
+
+
+def test_scale_delta() -> None:
+    spec = InterventionSpec('scale', 2.0)
+    assert spec.delta_activation(3.5) == 3.5
+    assert InterventionSpec('scale', 0.0).delta_activation(3.5) == -3.5
+
+
+def test_injection_delta() -> None:
+    spec = InterventionSpec('inject', -4.0)
+    assert spec.delta_activation(123.0) == -4.0
+
+
+def test_residual_delta_is_decoder_direction_times_coefficient_delta() -> None:
+    direction = torch.tensor([1.0, 2.0, -1.0])
+    delta = residual_delta(direction, 3.0, InterventionSpec('ablate', 0.0))
+    assert torch.allclose(delta, torch.tensor([-3.0, -6.0, 3.0]))
+
+
+def test_random_control_matches_norm_and_is_deterministic() -> None:
+    delta = torch.tensor([3.0, 4.0, 0.0])
+    a = normalized_random_control(delta, seed=7)
+    b = normalized_random_control(delta, seed=7)
+    assert torch.allclose(a, b)
+    assert torch.allclose(torch.linalg.vector_norm(a), torch.linalg.vector_norm(delta), atol=1e-5)

tests/test_metrics.py

@@ -0,0 +1,22 @@
+from __future__ import annotations
+
+import torch
+
+from featurelens.metrics import js_divergence_from_logits, reconstruction_metrics, safe_log_probability
+
+
+def test_reconstruction_metrics_perfect_match() -> None:
+    x = torch.tensor([1.0, 2.0, 3.0])
+    metrics = reconstruction_metrics(x, x.clone())
+    assert metrics['mse'] == 0.0
+    assert metrics['nmse'] == 0.0
+    assert abs(metrics['cosine'] - 1.0) < 1e-6
+
+
+def test_js_divergence_zero_for_identical_logits() -> None:
+    logits = torch.tensor([1.0, 2.0, -1.0])
+    assert abs(js_divergence_from_logits(logits, logits)) < 1e-8
+
+
+def test_safe_log_probability_is_finite_at_zero() -> None:
+    assert safe_log_probability(0.0) < 0.0

tests/test_sae.py

@@ -0,0 +1,57 @@
+from __future__ import annotations
+
+import torch
+
+from featurelens.sae import SAEWeights
+
+
+def make_sae() -> SAEWeights:
+    w_enc_t = torch.tensor(
+        [
+            [1.0, 0.0, -1.0, 0.5, 0.0],
+            [0.0, 1.0, 0.0, 0.5, -1.0],
+            [0.0, 0.0, 1.0, 0.0, 1.0],
+        ]
+    )
+    w_dec = torch.tensor(
+        [
+            [1.0, 0.0, 0.0, 0.5, 0.0],
+            [0.0, 1.0, 0.0, 0.5, 0.0],
+            [0.0, 0.0, 1.0, 0.0, 1.0],
+        ]
+    )
+    return SAEWeights(
+        layer=0,
+        w_enc_t=w_enc_t,
+        w_dec=w_dec,
+        b_enc=torch.zeros(5),
+        b_dec=torch.zeros(3),
+        top_k=2,
+    )
+
+
+def test_encode_keeps_topk_relu_features() -> None:
+    sae = make_sae()
+    encoding = sae.encode(torch.tensor([2.0, 1.0, -1.0]))
+    assert encoding.indices.shape == (2,)
+    assert set(encoding.indices.tolist()) == {0, 3}
+    assert encoding.active_count == 2
+    assert encoding.activation_for(0) == 2.0
+    assert encoding.activation_for(4) == 0.0
+
+
+def test_decode_sparse_uses_selected_decoder_columns() -> None:
+    sae = make_sae()
+    encoding = sae.encode(torch.tensor([2.0, 1.0, -1.0]))
+    reconstructed = sae.decode_sparse(encoding)
+    assert reconstructed.shape == (3,)
+    assert torch.isfinite(reconstructed).all()
+
+
+def test_batched_encode_and_decode() -> None:
+    sae = make_sae()
+    hidden = torch.tensor([[2.0, 1.0, -1.0], [0.0, 2.0, 2.0]])
+    encoding = sae.encode(hidden)
+    decoded = sae.decode_sparse(encoding)
+    assert encoding.indices.shape == (2, 2)
+    assert decoded.shape == hidden.shape

tests/test_split.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+
+from experiments.split import grouped_concept_split
+
+
+def test_grouped_split_keeps_paraphrase_pairs_together() -> None:
+    rows = []
+    for concept in ('a', 'b'):
+        for pair in range(4):
+            for variant in range(2):
+                rows.append(
+                    {
+                        'concept': concept,
+                        'pair_id': f'{concept}-{pair}',
+                        'variant': variant,
+                    }
+                )
+    train, test = grouped_concept_split(rows, test_fraction=0.25, seed=42)
+    train_pairs = {rows[idx]['pair_id'] for idx in train}
+    test_pairs = {rows[idx]['pair_id'] for idx in test}
+    assert train_pairs.isdisjoint(test_pairs)
+    assert {rows[idx]['concept'] for idx in train} == {'a', 'b'}
+    assert {rows[idx]['concept'] for idx in test} == {'a', 'b'}