File size: 12,023 Bytes
498b9c9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
---
license: other
base_model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
tags: [interpretability, jacobian-lens, dory]
---

# Dory implementation: forward pass and Jacobian lens

A plain-PyTorch forward pass of NVIDIA's looped language model
[Dory1.0-4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) (arXiv 2611.03998) with a hook at
every sublayer and loop boundary, and a plain Jacobian lens (J-lens) fitted on it.

**Status, 2026-11-16: first fits, exploratory.** The estimator is verified. Plain J-lenses are fitted for 1, 2, 3
and 4 loops, and R-lenses and J++ lenses for 1 and 2 loops; all lens files are under `lenses/`. Read "Which lens to
use" and "What the lens is like" before relying on a readout. A copy of the code and records, without the lens
files, is in `conditional-model-diffing/dory_implementation`.

## Files

| Path | What it is |
|---|---|
| `dory/model.py` | The model: loads the official checkpoint, runs any number of loops, exposes every residual state through a `tap(name, h)` callback. Also `load_tokenizer`. |
| `dory/jlens.py` | The J-lens: the per-prompt Jacobian estimator, the `JLens` class (load, transport, readout) and a logit-lens baseline. |
| `dory/jpp.py`, `dory/lens_eval.py` | J++ pieces (activation router, expert weights) and the readout evaluation loader. The LRP backward rules are in `dory/model.py` (`with lrp_rules(): ...`). |
| `dory/chain.py` | A model-driven roll-out tried as a chain-lens analogue. It did not work; kept for the record. |
| `lenses/jlens_k{1,2,3,4}.safetensors` | Plain J-lenses for 1 to 4 loops, float32 matrices of 2560 x 2560 (1.9 to 5.1 GB). |
| `lenses/rlens_k{1,2}.safetensors`, `lenses/jpp_k{1,2}.safetensors` | R-lenses and J++ lenses for 1 and 2 loops (1.9 and 3.0 GB each). J++ is the default. |
| `lenses/router_and_weights_k{1,2}.pt` | The J++ activation router and expert weights (torch pickle). |
| `fit_records/<run>/` | For each fit: `metrics.json` (per-node noise and early-position numbers), `readouts.txt` (tables and example readouts), `prompts.json`, `run.json`. |
| `as_run/` | The fitting script, the readout example and the estimator test exactly as they were run. Their import paths are those of the fitting project, not of this repo. |

## Use

```python
import sys, torch
from huggingface_hub import snapshot_download, hf_hub_download
repo = snapshot_download("senku21x/dory_implementation", allow_patterns=["dory/*"])
sys.path.insert(0, repo)
from dory.model import Dory, load_tokenizer
from dory.jlens import JLens, logit_lens, node_key

snap = snapshot_download("nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16")
tok = load_tokenizer(snap)                       # AutoTokenizer fails: transformers has no `dory` config class yet
model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32)
lens = JLens.load(hf_hub_download("senku21x/dory_implementation", "lenses/jpp_k2.safetensors"), device="cuda")

ids = tok("The capital of the country where sushi originated is", return_tensors="pt").input_ids.cuda()
states = {}
with torch.no_grad():
    final = model.forward(ids, n_loops=lens.k, tap=lambda name, h: states.__setitem__(name, h))
for node in lens.nodes():
    logits = lens.logits(model, states[node][0, -1], node)       # unembed(l2norm(J @ h))
    print(node_key(node), [tok.decode([t]) for t in logits.topk(5).indices.tolist()])
```

A lens is fitted for one loop count; run the model with `n_loops = lens.k`.

## The model, and what a "layer" is

Dory's checkpoint counts sublayers: 72 of them, alternating attention and MLP. Sublayers 0 to 15 are the input
block (run once), 16 to 55 the recurrent block (run once per loop, weights shared), 56 to 71 the output block (run
once). With `x` the input block's output and `N` the L2 norm, loop `t` computes `u_t = N(s_{t-1} + x)` and
`s_t = N(F(u_t))`, starting from `s_0 = x`. Every state is a unit vector.

Lens source nodes, in forward order. Every sublayer output has exactly one node.

| Node | State |
|---|---|
| `emb` | token embedding |
| `in.00` ... `in.14` | after input sublayers 0 to 14 |
| `x` | after sublayer 15, the input block's output |
| `u.t` | input to loop `t` |
| `rec.t.16` ... `rec.t.54` | after recurrent sublayers 16 to 54 in loop `t` |
| `s.t` | after sublayer 55 in loop `t`, the loop's output |
| `out.56` ... `out.70` | after output sublayers 56 to 70 |

The target is the final state (after sublayer 71), which the LM head reads. In `model.forward` the tap names are
tuples, for example `("rec", 2, 41)`; `node_key` turns them into the strings above.

## How the lens was fitted

The estimator of Gurnee et al. 2026 ("Verbalizable Representations Form a Global Workspace in Language Models"):
for each source node, the Jacobian of the final state with respect to the node's state, summed over target
positions at or after the source position, averaged over source positions and then over prompts. One backward pass
per output dimension (2,560 per prompt, batched).

- 50 prompts: the first 128 tokens of 50 WikiText-103 paragraphs (validation split, seed 0). No BOS token; the
  tokenizer adds none.
- The first 16 source positions are left out of the average.
- float32 weights with TF32 matrix multiplies. 39 seconds per prompt at 2 loops on one H200.
- Each lens file also holds the mean state of every node over the fitting positions (`lens.mean_state`), for
  centring experiments.

## Checks

- **Estimator.** On the real model, the autograd Jacobian sums match central finite differences at 10 nodes and
  all three position ranges (cosine 0.9998 or higher, norm ratio within 1%). TF32 against exact float32 on the same
  rows: cosine at least 0.9999, relative error at most 1.4%.
- **Forward pass.** All 507 checkpoint tensors load. Greedy text is coherent at 1, 2, 4 and 8 loops; loss on
  ordinary prose is about 2.0 nats per token (7.8 with the tokens shuffled); the output block applied to a loop-end
  state reproduces the model run with that many loops exactly. The code was read step by step against the authors'
  transformers pull request (50099) and matches it.
- **Not done.** No number-for-number comparison with the authors' implementation. The sliding-window attention
  (1,024 tokens) is exercised on a long document but not validated.

## Which lens to use

Median split-half cosine by block, for each fitted lens (1 means the lens fitted on the 25 even-numbered prompts
agrees with the one fitted on the 25 odd-numbered prompts):

| File | Loops | Matrices | Size | Input block | Loop 1 | Loop 2 | Loop 3 | Loop 4 | Output block |
|---|---|---|---|---|---|---|---|---|---|
| `jlens_k1.safetensors` | 1 | 73 | 1.9 GB | 0.80 | 0.97 |  |  |  | 1.00 |
| `jlens_k2.safetensors` | 2 | 114 | 3.0 GB | 0.52 | 0.29 | 0.97 |  |  | 1.00 |
| `jlens_k3.safetensors` | 3 | 155 | 4.1 GB | 0.03 | 0.01 | 0.65 | 0.97 |  | 1.00 |
| `jlens_k4.safetensors` | 4 | 196 | 5.1 GB | 0.09 | -0.00 | 0.12 | 0.75 | 0.97 | 1.00 |

- **A lens aimed at the final state reaches back about one loop.** Each lens is stable for its last loop and the
  output block, and noise before that.
- **So read loop `t` through the lens fitted at `t` loops.** The states of loops 1 to `t` in a longer run are the
  same as in a `t`-loop run, and Dory is trained so that any loop end can go straight to the output block. On 17
  hand-written two-hop prompts (4-loop run, punctuation-like tokens dropped), the intermediate reached the top 10
  at some node of loops 1 / 2 / 3 / 4 on 41 / 47 / 47 / 47% of prompts read this way, against 0 / 0 / 24 / 47%
  through the 4-loop lens alone. Exploratory: small hand-made set, best rank over 41 nodes per loop.
- **The per-loop lenses are almost the same matrix.** At the same sublayer, the loop-`t` matrix of the `t`-loop
  lens has cosine 0.995 or more with the loop-1 matrix of the 1-loop lens (0.98 at the first recurrent sublayer),
  and the output-block matrices agree to 1.000. In practice `jlens_k1.safetensors` alone serves every loop: use its
  `rec.1.i` matrix for `rec.t.i`, its `s.1` for `s.t`, and its `out.i` for the output block.
- **The states are almost the same too.** At the same sublayer, consecutive loops' states have cosine 0.95 to 0.99
  between loops 1 and 2 and 0.999 after that (3 prompts), so the readouts of different loops are nearly identical.
  Whatever a later loop changes is in a very small part of the state that a readout of the whole state does not
  isolate.

## What the lens is like (2 loops, 50 prompts)

Numbers are from `fit_records/k2_n50_skip16_s0/`; held-out figures are from 8 paragraphs not used in the fit.

| Block | Nodes | Split-half cosine (median) | Top-1 agreement with the model, J-lens | Logit lens | Punctuation-like share of top 10 |
|---|---|---|---|---|---|
| Input block | 17 | 0.52 | 0.00 | 0.00 | 0.66 |
| Loop 1 | 41 | 0.29 | 0.01 | 0.00 | 0.30 |
| Loop 2 | 41 | 0.97 | 0.04 | 0.00 | 0.21 |
| Output block | 15 | 1.00 | 0.41 | 0.29 | 0.10 |

"Split-half cosine" compares the lens fitted on the 25 even-numbered prompts with the one fitted on the 25
odd-numbered prompts; 1 means the two agree.

- **The input block and loop 1 are not converged.** Their matrices are mostly prompt-to-prompt noise at 50 prompts
  (split-half 0.02 to 0.6; it passes 0.8 only from `rec.1.51`). Treat readouts there as unreliable. Loop 2 and the
  output block are stable.
- **Readouts are dominated by punctuation and formatting tokens** until the output block. On four two-hop prompts
  the unspoken intermediate never reached the top 10 before the output block (best ranks in loop 2: 30 to 373). The
  model itself did not answer two of the four directly, so this is not a clean test.
- **The logit lens fails at loop boundaries**: `s.t` and `u.t` are close to one fixed vector, and the unembedding
  reads the same few tokens whatever the prompt.
- **Early positions.** No sign of a first-four-tokens problem. Every state in Dory is a unit vector, so there are no
  high-norm tokens. Per-position Jacobians are about 3.7 times larger at position 0 and about 2 times larger at
  positions 1 to 15 than later (partly because earlier positions have more target positions to sum over). Leaving
  out 0, 4 or 16 positions gives nearly the same loop-2 and output-block matrices (cosine 0.99 or higher) and
  moderately different input-block ones (0.92).

## R-lens and J++ (1 and 2 loops, 64 prompts)

Recall@10 of the labelled intermediate on a held-out half of the Gurnee et al. readout sets (multihop,
multilingual, typo, association, poetry; 181 test items), at any of seven nodes equally spaced over the unrolled
depth, macro over tasks, punctuation-like tokens dropped. Brackets: 95% bootstrap interval over test items.

| Lens | 1 loop | 2 loops |
|---|---|---|
| Logit lens | 0.244 [0.18, 0.31] | 0.189 [0.14, 0.25] |
| Plain J-lens | 0.319 [0.26, 0.39] | 0.268 [0.22, 0.33] |
| R-lens | 0.396 [0.33, 0.46] | 0.321 [0.26, 0.39] |
| J++ | 0.465 [0.40, 0.53] | 0.423 [0.36, 0.49] |
| J++, weights learned on shuffled labels | 0.238 | 0.225 |

- J++ minus R-lens: +0.069 [+0.03, +0.12] at 1 loop, +0.102 [+0.02, +0.18] at 2 loops.
- With the LRP rules the lens is stable in every block (split-half cosine 0.99), including loop 1 of a 2-loop run.
- Limits: the J++ weights were learned on the other half of the same five tasks; no swap or ablation test has
  been run; multihop has 10 or 11 test items because Dory answers about a quarter of them; one seed, one router,
  one split.
- LRP choices specific to Dory (not covered by the R-lens or J++ posts): every L2 norm's denominator is constant
  except the query and key norms, and the attention output gate is treated as a constant.
- Records: `fit_records/jpp_k1_n64_e8_s0_r2/` and `fit_records/jpp_k2_n64_e8_s0/`. Lens files: `lenses/jpp_k<k>.safetensors` and `lenses/rlens_k<k>.safetensors`.

## Not here yet

A centred readout, causal tests of the lens directions, and a multi-token readout (Dory has no multi-token
prediction head, so the chain lens does not apply directly).