docs/COLAB.md

Running the FeatureLens offline study on Google Colab

This guide is for the offline empirical study, not the public Hugging Face Space. The Space remains the interactive demo; Colab is only used to produce the study artifacts once.

The easiest route is the included notebook:

notebooks/FeatureLens_Offline_Study_Colab.ipynb

What the run does

The full study executes:

1. build/verify the 224-prompt discovery set and 28 causal tasks;
2. collect Qwen3 residuals and prompt-wide Qwen-Scope SAE activations at layers 4, 14, and 26;
3. fit/evaluate held-out SAE-feature classifiers and the dense residual probe;
4. run single-feature random-controlled causal interventions;
5. run 1/3/5 feature-set interventions and controls;
6. compute candidate-selection stability, study synthesis, figures, and report;
7. validate that the publishable artifacts are complete and methodologically compatible.

Recommended runtime

Use an NVIDIA GPU runtime. A 16 GB T4 is the practical baseline; an L4 or A100 gives more headroom. The notebook detects GPU memory and defaults activation collection to batch size 8 below 20 GB VRAM and 16 otherwise.

Runtime varies with Colab allocation, downloads, and prompt lengths. For planning, budget roughly 1–2 hours on a T4 for a first complete run, including model/SAE downloads; faster GPUs can be substantially quicker. Treat this as a planning estimate, not a benchmark.

Persistence model

Colab VMs are temporary. The notebook mounts Google Drive and replaces the repo's artifacts/ directory with a symlink to a Drive-backed directory. This means:

• activation artifacts survive a runtime reset;
• task-level causal and feature-set checkpoints survive a reset;
• python -m experiments.run_all --resume can continue rather than restart completed work;
• Hugging Face model caches remain on the Colab VM for speed and may need to be downloaded again after a new runtime.

The causal and feature-set stages checkpoint after every completed task in v0.15. Completion-marker files are runtime bookkeeping and are excluded from the publishable bundle.

Notebook configuration

At the top of the notebook set:

REPO_URL = "PASTE_YOUR_GIT_REPO_URL_HERE"
BRANCH = "main"
DRIVE_RUN_NAME = "FeatureLens_offline_v015"

REPO_URL can be the public Git URL of the Hugging Face Space repository or another Git mirror containing the same FeatureLens source.

If your default branch is not main, change BRANCH.

If activation collection runs out of memory

The notebook chooses a conservative batch size automatically. If CUDA still runs out of memory, rerun the pipeline cell with:

python -m experiments.run_all --resume --activation-batch-size 4

Changing the activation batch size changes memory/time trade-offs, not the experiment definition.

After the run

The notebook runs:

python -m scripts.validate_artifacts

and creates a small archive containing only the publishable study outputs. It excludes:

• artifacts/activations/;
• .complete checkpoint markers;
• model/SAE caches;
• temporary files.

Extract the archive over the local FeatureLens repository so the files land under artifacts/, then run the normal local release checks and push. The public Study tab will read those measured artifacts automatically.

Publishable outputs

The bundle is expected to contain files such as:

• artifacts/feature_catalog.csv
• artifacts/layer_metrics.csv
• artifacts/stability.csv
• artifacts/selection_stability.csv
• artifacts/causal_results_final_token.csv
• artifacts/causal_results_max_active.csv
• artifacts/causal_position_summary.csv
• artifacts/feature_set_results.csv
• artifacts/study_feature_summary.csv
• artifacts/study_summary.json
• artifacts/summary.json
• artifacts/report.md
• artifacts/figures/*.png

Do not commit the artifacts/activations/ directory.

v0.16 causal addendum after a completed v0.15 study

If the full v0.15 Colab study already completed, do not rerun the full notebook. Use:

notebooks/FeatureLens_Causal_Addendum_Colab.ipynb

The addendum notebook copies only the small existing study outputs to a new Drive folder, preserves the final-token causal baseline, computes the 28-task max_feature_activation causal policy, regenerates the CPU study/report artifacts, validates them, and creates a new publishable ZIP.

The addendum does not recollect 224-prompt activations, refit feature/probe evaluations, rerun stability resampling, or rerun the 1/3/5 feature-set stage.