movingcastles/zero-jlens

J-lenses for Zero and Qwen3-8B-Base

Jacobian lenses fitted for these checkpoints:

• Zero: movingcastles/zero, revision 856f7d4b8d98088b2f833c9e69a08f286ceb3629.
• Base: Qwen/Qwen3-8B-Base, revision 49e3418fbbbca6ecbdf9608b4d22e5a407081db4.

Zero was formerly named movingcastles/zero-rl-v2; the lens paths retain that name. Use each lens with its own checkpoint.

Files

Lens	Checkpoint	Fitting corpus
zero-rl-v2/fineweb200/lens.pt	Zero	200 FineWeb texts
zero-rl-v2/sft200/lens.pt	Zero	200 Zero SFT conversations
zero-rl-v2/chinese200/lens.pt	Zero	200 Chinese Wikipedia articles
qwen3-8b-base/fineweb200/lens.pt	Base	The same FineWeb texts
qwen3-8b-base/sft200/lens.pt	Base	The same SFT conversations

Each lens contains 34 matrices of shape 4096 × 4096. A matrix maps the residual after its source block (0–33) into the basis of the residual after block 34. Apply the checkpoint's final RMSNorm and unembedding to get a token reading. Block 34 is read directly; block 35 gives the model's actual next-token distribution.

Files use JacobianLens.save format: fitted in fp32, stored in fp16 and loaded as fp32. MANIFEST.json records each file's SHA-256 and its fp32 source's SHA-256. Pin the model and lens repository revisions when recording a run.

Fitting

Each corpus was sampled without replacement. We used the first 128 tokens of each document and excluded shorter documents:

• FineWeb: HuggingFaceFW/fineweb, sample-10BT, train, revision 9bb295dd; deduplicated by URL and content before sampling.
• SFT: movingcastles/zero-prod-6k, train.jsonl, revision 45ec7733; rendered with Zero's chat template. Access is gated. Later dataset revisions use data/train.parquet. Other language models wrote the user turns, which make up most fitted positions.
• Chinese Wikipedia: wikimedia/wikipedia, 20231101.zh, train, revision b04c8d1c.

Fitting uses the reference implementation at commit 581d398613e5602a5af361e1c34d3a92ea82ba8e. FineWeb and Wikipedia use raw text; no BOS token is added. For each source position from 16 to 126, sum the Jacobians to block-34 positions at or after it, then average over source positions and equally over documents. Computation uses fp32 and RoPE theta 1e6.

Use

pip install -r examples/requirements.txt
python examples/read_prompt.py --model movingcastles/zero --lens zero-rl-v2/sft200 --prompt "who are you?"

The script applies the chat template and prints the top tokens at each requested layer. At the final block, it prints the model's next-token distribution at the selected position.

--position selects where to read: reply-start, im_start, assistant, or a token index.

--raw skips the chat template; use it for plain text or Base. --layers 8,16,24 selects layers, and --lens-repo <dir> reads lenses from a local copy. --model also accepts a local checkpoint path.

Load Zero with RoPE theta 1e6, as used for fitting. Its pinned config contains both legacy rope_theta: 10000 and rope_parameters.rope_theta: 1000000; check that your loader uses 1e6.

Limits

• The fitting corpus affects the reading: Zero's SFT and FineWeb lenses can give different words for the same activation.
• Fitting covers only the first 128 tokens. Readings far beyond that window are extrapolations, and early-layer readings are noisy.
• A reading is not a transcript of the model's thoughts. A token's probability under the lens is not the probability that an interpretation is true.