agu18dec/oracle-lens

Oracle Lens — Qwen3.6-27B

An oracle lens reads a model's internal activations by injecting an activation at a fixed slot and having a LoRA-tuned copy emit the natural-language phrases that activation encodes. Unlike a cosine/logit lens (one vocabulary token per position), it emits multi-token English phrases, so it surfaces whole latent concepts and intermediates — e.g. reading New Zealand, Statue of Liberty, IndexError, or a latent Adam Smith the model never says out loud.

Each folder is a LoRA adapter (r=16, alpha=32, all-linear). To use: load the adapter on Qwen3.6-27B, render the prompt Activation: @\nList the top K phrases of exactly N tokens encoded by this activation, one per line, most important first., and splice the (whitened-space or raw, per recipe) activation into the @ slot at the lens's injection layer.

Lenses

lens	recipe	held-out GT-FVE (K=8, teacher ceiling 0.483)
qwen3.6-27b-l44-add-100k	layer-44, ADD injection (alpha=1), 100k teacher labels, 2 epochs — THE signs-of-life lens used in every eval	0.183
qwen3.6-27b-l44-add-10k	layer-44 add, 10k labels	0.157
qwen3.6-27b-l44-add-30k	layer-44 add, 30k labels	0.154
qwen3.6-27b-l44-add-200k	layer-44 add, 200k labels	0.179
qwen3.6-27b-l1-replace-100k	layer-1 REPLACE (scale 1.0), 100k labels — best single-layer recipe	0.188
qwen3.6-27b-l1-replace-30k	layer-1 replace, 30k labels	0.172
qwen3.6-27b-multilayer-l20-60	ONE lens reading layers 20-60 (every 4th), layer-1 replace, per-layer magnitude normalized, 20k rows/layer	0.178 mean

Start with qwen3.6-27b-l44-add-100k — the checkpoint used in all the reported evals.

Injection sites: l44-add = add the raw activation to the layer-44 residual at the slot; l1-replace = overwrite the layer-1 residual at the slot (best single-layer, and the site the multi-layer lens uses). FVE is measured in whitened space against ground-truth held-out activations, K-matched to the phrase budget.

Pipeline + eval code: global_workspace.oracle_lens (commit 870658a).