File size: 12,023 Bytes
498b9c9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 | ---
license: other
base_model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
tags: [interpretability, jacobian-lens, dory]
---
# Dory implementation: forward pass and Jacobian lens
A plain-PyTorch forward pass of NVIDIA's looped language model
[Dory1.0-4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) (arXiv 2611.03998) with a hook at
every sublayer and loop boundary, and a plain Jacobian lens (J-lens) fitted on it.
**Status, 2026-11-16: first fits, exploratory.** The estimator is verified. Plain J-lenses are fitted for 1, 2, 3
and 4 loops, and R-lenses and J++ lenses for 1 and 2 loops; all lens files are under `lenses/`. Read "Which lens to
use" and "What the lens is like" before relying on a readout. A copy of the code and records, without the lens
files, is in `conditional-model-diffing/dory_implementation`.
## Files
| Path | What it is |
|---|---|
| `dory/model.py` | The model: loads the official checkpoint, runs any number of loops, exposes every residual state through a `tap(name, h)` callback. Also `load_tokenizer`. |
| `dory/jlens.py` | The J-lens: the per-prompt Jacobian estimator, the `JLens` class (load, transport, readout) and a logit-lens baseline. |
| `dory/jpp.py`, `dory/lens_eval.py` | J++ pieces (activation router, expert weights) and the readout evaluation loader. The LRP backward rules are in `dory/model.py` (`with lrp_rules(): ...`). |
| `dory/chain.py` | A model-driven roll-out tried as a chain-lens analogue. It did not work; kept for the record. |
| `lenses/jlens_k{1,2,3,4}.safetensors` | Plain J-lenses for 1 to 4 loops, float32 matrices of 2560 x 2560 (1.9 to 5.1 GB). |
| `lenses/rlens_k{1,2}.safetensors`, `lenses/jpp_k{1,2}.safetensors` | R-lenses and J++ lenses for 1 and 2 loops (1.9 and 3.0 GB each). J++ is the default. |
| `lenses/router_and_weights_k{1,2}.pt` | The J++ activation router and expert weights (torch pickle). |
| `fit_records/<run>/` | For each fit: `metrics.json` (per-node noise and early-position numbers), `readouts.txt` (tables and example readouts), `prompts.json`, `run.json`. |
| `as_run/` | The fitting script, the readout example and the estimator test exactly as they were run. Their import paths are those of the fitting project, not of this repo. |
## Use
```python
import sys, torch
from huggingface_hub import snapshot_download, hf_hub_download
repo = snapshot_download("senku21x/dory_implementation", allow_patterns=["dory/*"])
sys.path.insert(0, repo)
from dory.model import Dory, load_tokenizer
from dory.jlens import JLens, logit_lens, node_key
snap = snapshot_download("nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16")
tok = load_tokenizer(snap) # AutoTokenizer fails: transformers has no `dory` config class yet
model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
lens = JLens.load(hf_hub_download("senku21x/dory_implementation", "lenses/jpp_k2.safetensors"), device="cuda")
ids = tok("The capital of the country where sushi originated is", return_tensors="pt").input_ids.cuda()
states = {}
with torch.no_grad():
final = model.forward(ids, n_loops=lens.k, tap=lambda name, h: states.__setitem__(name, h))
for node in lens.nodes():
logits = lens.logits(model, states[node][0, -1], node) # unembed(l2norm(J @ h))
print(node_key(node), [tok.decode([t]) for t in logits.topk(5).indices.tolist()])
```
A lens is fitted for one loop count; run the model with `n_loops = lens.k`.
## The model, and what a "layer" is
Dory's checkpoint counts sublayers: 72 of them, alternating attention and MLP. Sublayers 0 to 15 are the input
block (run once), 16 to 55 the recurrent block (run once per loop, weights shared), 56 to 71 the output block (run
once). With `x` the input block's output and `N` the L2 norm, loop `t` computes `u_t = N(s_{t-1} + x)` and
`s_t = N(F(u_t))`, starting from `s_0 = x`. Every state is a unit vector.
Lens source nodes, in forward order. Every sublayer output has exactly one node.
| Node | State |
|---|---|
| `emb` | token embedding |
| `in.00` ... `in.14` | after input sublayers 0 to 14 |
| `x` | after sublayer 15, the input block's output |
| `u.t` | input to loop `t` |
| `rec.t.16` ... `rec.t.54` | after recurrent sublayers 16 to 54 in loop `t` |
| `s.t` | after sublayer 55 in loop `t`, the loop's output |
| `out.56` ... `out.70` | after output sublayers 56 to 70 |
The target is the final state (after sublayer 71), which the LM head reads. In `model.forward` the tap names are
tuples, for example `("rec", 2, 41)`; `node_key` turns them into the strings above.
## How the lens was fitted
The estimator of Gurnee et al. 2026 ("Verbalizable Representations Form a Global Workspace in Language Models"):
for each source node, the Jacobian of the final state with respect to the node's state, summed over target
positions at or after the source position, averaged over source positions and then over prompts. One backward pass
per output dimension (2,560 per prompt, batched).
- 50 prompts: the first 128 tokens of 50 WikiText-103 paragraphs (validation split, seed 0). No BOS token; the
tokenizer adds none.
- The first 16 source positions are left out of the average.
- float32 weights with TF32 matrix multiplies. 39 seconds per prompt at 2 loops on one H200.
- Each lens file also holds the mean state of every node over the fitting positions (`lens.mean_state`), for
centring experiments.
## Checks
- **Estimator.** On the real model, the autograd Jacobian sums match central finite differences at 10 nodes and
all three position ranges (cosine 0.9998 or higher, norm ratio within 1%). TF32 against exact float32 on the same
rows: cosine at least 0.9999, relative error at most 1.4%.
- **Forward pass.** All 507 checkpoint tensors load. Greedy text is coherent at 1, 2, 4 and 8 loops; loss on
ordinary prose is about 2.0 nats per token (7.8 with the tokens shuffled); the output block applied to a loop-end
state reproduces the model run with that many loops exactly. The code was read step by step against the authors'
transformers pull request (50099) and matches it.
- **Not done.** No number-for-number comparison with the authors' implementation. The sliding-window attention
(1,024 tokens) is exercised on a long document but not validated.
## Which lens to use
Median split-half cosine by block, for each fitted lens (1 means the lens fitted on the 25 even-numbered prompts
agrees with the one fitted on the 25 odd-numbered prompts):
| File | Loops | Matrices | Size | Input block | Loop 1 | Loop 2 | Loop 3 | Loop 4 | Output block |
|---|---|---|---|---|---|---|---|---|---|
| `jlens_k1.safetensors` | 1 | 73 | 1.9 GB | 0.80 | 0.97 | | | | 1.00 |
| `jlens_k2.safetensors` | 2 | 114 | 3.0 GB | 0.52 | 0.29 | 0.97 | | | 1.00 |
| `jlens_k3.safetensors` | 3 | 155 | 4.1 GB | 0.03 | 0.01 | 0.65 | 0.97 | | 1.00 |
| `jlens_k4.safetensors` | 4 | 196 | 5.1 GB | 0.09 | -0.00 | 0.12 | 0.75 | 0.97 | 1.00 |
- **A lens aimed at the final state reaches back about one loop.** Each lens is stable for its last loop and the
output block, and noise before that.
- **So read loop `t` through the lens fitted at `t` loops.** The states of loops 1 to `t` in a longer run are the
same as in a `t`-loop run, and Dory is trained so that any loop end can go straight to the output block. On 17
hand-written two-hop prompts (4-loop run, punctuation-like tokens dropped), the intermediate reached the top 10
at some node of loops 1 / 2 / 3 / 4 on 41 / 47 / 47 / 47% of prompts read this way, against 0 / 0 / 24 / 47%
through the 4-loop lens alone. Exploratory: small hand-made set, best rank over 41 nodes per loop.
- **The per-loop lenses are almost the same matrix.** At the same sublayer, the loop-`t` matrix of the `t`-loop
lens has cosine 0.995 or more with the loop-1 matrix of the 1-loop lens (0.98 at the first recurrent sublayer),
and the output-block matrices agree to 1.000. In practice `jlens_k1.safetensors` alone serves every loop: use its
`rec.1.i` matrix for `rec.t.i`, its `s.1` for `s.t`, and its `out.i` for the output block.
- **The states are almost the same too.** At the same sublayer, consecutive loops' states have cosine 0.95 to 0.99
between loops 1 and 2 and 0.999 after that (3 prompts), so the readouts of different loops are nearly identical.
Whatever a later loop changes is in a very small part of the state that a readout of the whole state does not
isolate.
## What the lens is like (2 loops, 50 prompts)
Numbers are from `fit_records/k2_n50_skip16_s0/`; held-out figures are from 8 paragraphs not used in the fit.
| Block | Nodes | Split-half cosine (median) | Top-1 agreement with the model, J-lens | Logit lens | Punctuation-like share of top 10 |
|---|---|---|---|---|---|
| Input block | 17 | 0.52 | 0.00 | 0.00 | 0.66 |
| Loop 1 | 41 | 0.29 | 0.01 | 0.00 | 0.30 |
| Loop 2 | 41 | 0.97 | 0.04 | 0.00 | 0.21 |
| Output block | 15 | 1.00 | 0.41 | 0.29 | 0.10 |
"Split-half cosine" compares the lens fitted on the 25 even-numbered prompts with the one fitted on the 25
odd-numbered prompts; 1 means the two agree.
- **The input block and loop 1 are not converged.** Their matrices are mostly prompt-to-prompt noise at 50 prompts
(split-half 0.02 to 0.6; it passes 0.8 only from `rec.1.51`). Treat readouts there as unreliable. Loop 2 and the
output block are stable.
- **Readouts are dominated by punctuation and formatting tokens** until the output block. On four two-hop prompts
the unspoken intermediate never reached the top 10 before the output block (best ranks in loop 2: 30 to 373). The
model itself did not answer two of the four directly, so this is not a clean test.
- **The logit lens fails at loop boundaries**: `s.t` and `u.t` are close to one fixed vector, and the unembedding
reads the same few tokens whatever the prompt.
- **Early positions.** No sign of a first-four-tokens problem. Every state in Dory is a unit vector, so there are no
high-norm tokens. Per-position Jacobians are about 3.7 times larger at position 0 and about 2 times larger at
positions 1 to 15 than later (partly because earlier positions have more target positions to sum over). Leaving
out 0, 4 or 16 positions gives nearly the same loop-2 and output-block matrices (cosine 0.99 or higher) and
moderately different input-block ones (0.92).
## R-lens and J++ (1 and 2 loops, 64 prompts)
Recall@10 of the labelled intermediate on a held-out half of the Gurnee et al. readout sets (multihop,
multilingual, typo, association, poetry; 181 test items), at any of seven nodes equally spaced over the unrolled
depth, macro over tasks, punctuation-like tokens dropped. Brackets: 95% bootstrap interval over test items.
| Lens | 1 loop | 2 loops |
|---|---|---|
| Logit lens | 0.244 [0.18, 0.31] | 0.189 [0.14, 0.25] |
| Plain J-lens | 0.319 [0.26, 0.39] | 0.268 [0.22, 0.33] |
| R-lens | 0.396 [0.33, 0.46] | 0.321 [0.26, 0.39] |
| J++ | 0.465 [0.40, 0.53] | 0.423 [0.36, 0.49] |
| J++, weights learned on shuffled labels | 0.238 | 0.225 |
- J++ minus R-lens: +0.069 [+0.03, +0.12] at 1 loop, +0.102 [+0.02, +0.18] at 2 loops.
- With the LRP rules the lens is stable in every block (split-half cosine 0.99), including loop 1 of a 2-loop run.
- Limits: the J++ weights were learned on the other half of the same five tasks; no swap or ablation test has
been run; multihop has 10 or 11 test items because Dory answers about a quarter of them; one seed, one router,
one split.
- LRP choices specific to Dory (not covered by the R-lens or J++ posts): every L2 norm's denominator is constant
except the query and key norms, and the attention output gate is treated as a constant.
- Records: `fit_records/jpp_k1_n64_e8_s0_r2/` and `fit_records/jpp_k2_n64_e8_s0/`. Lens files: `lenses/jpp_k<k>.safetensors` and `lenses/rlens_k<k>.safetensors`.
## Not here yet
A centred readout, causal tests of the lens directions, and a multi-token readout (Dory has no multi-token
prediction head, so the chain lens does not apply directly).
|