|
Download README.md from senku21x/dory_implementation: direct link, hf CLI and curl.
- Browser
- Download file 12 kB
-
https://huggingface.co/senku21x/dory_implementation/resolve/main/README.md
- Command line
-
hf download hf://senku21x/dory_implementation/README.md
-
curl -L -o README.md https://huggingface.co/senku21x/dory_implementation/resolve/main/README.md
12 kB
| license: other | |
| base_model: nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 | |
| tags: [interpretability, jacobian-lens, dory] | |
| # Dory implementation: forward pass and Jacobian lens | |
| A plain-PyTorch forward pass of NVIDIA's looped language model | |
| [Dory1.0-4B](https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16) (arXiv 2611.03998) with a hook at | |
| every sublayer and loop boundary, and a plain Jacobian lens (J-lens) fitted on it. | |
| **Status, 2026-11-16: first fits, exploratory.** The estimator is verified. Plain J-lenses are fitted for 1, 2, 3 | |
| and 4 loops, and R-lenses and J++ lenses for 1 and 2 loops; all lens files are under `lenses/`. Read "Which lens to | |
| use" and "What the lens is like" before relying on a readout. A copy of the code and records, without the lens | |
| files, is in `conditional-model-diffing/dory_implementation`. | |
| ## Files | |
| | Path | What it is | | |
| |---|---| | |
| | `dory/model.py` | The model: loads the official checkpoint, runs any number of loops, exposes every residual state through a `tap(name, h)` callback. Also `load_tokenizer`. | | |
| | `dory/jlens.py` | The J-lens: the per-prompt Jacobian estimator, the `JLens` class (load, transport, readout) and a logit-lens baseline. | | |
| | `dory/jpp.py`, `dory/lens_eval.py` | J++ pieces (activation router, expert weights) and the readout evaluation loader. The LRP backward rules are in `dory/model.py` (`with lrp_rules(): ...`). | | |
| | `dory/chain.py` | A model-driven roll-out tried as a chain-lens analogue. It did not work; kept for the record. | | |
| | `lenses/jlens_k{1,2,3,4}.safetensors` | Plain J-lenses for 1 to 4 loops, float32 matrices of 2560 x 2560 (1.9 to 5.1 GB). | | |
| | `lenses/rlens_k{1,2}.safetensors`, `lenses/jpp_k{1,2}.safetensors` | R-lenses and J++ lenses for 1 and 2 loops (1.9 and 3.0 GB each). J++ is the default. | | |
| | `lenses/router_and_weights_k{1,2}.pt` | The J++ activation router and expert weights (torch pickle). | | |
| | `fit_records/<run>/` | For each fit: `metrics.json` (per-node noise and early-position numbers), `readouts.txt` (tables and example readouts), `prompts.json`, `run.json`. | | |
| | `as_run/` | The fitting script, the readout example and the estimator test exactly as they were run. Their import paths are those of the fitting project, not of this repo. | | |
| ## Use | |
| ```python | |
| import sys, torch | |
| from huggingface_hub import snapshot_download, hf_hub_download | |
| repo = snapshot_download("senku21x/dory_implementation", allow_patterns=["dory/*"]) | |
| sys.path.insert(0, repo) | |
| from dory.model import Dory, load_tokenizer | |
| from dory.jlens import JLens, logit_lens, node_key | |
| snap = snapshot_download("nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16") | |
| tok = load_tokenizer(snap) # AutoTokenizer fails: transformers has no `dory` config class yet | |
| model = Dory.from_snapshot(snap, device="cuda", dtype=torch.float32) | |
| lens = JLens.load(hf_hub_download("senku21x/dory_implementation", "lenses/jpp_k2.safetensors"), device="cuda") | |
| ids = tok("The capital of the country where sushi originated is", return_tensors="pt").input_ids.cuda() | |
| states = {} | |
| with torch.no_grad(): | |
| final = model.forward(ids, n_loops=lens.k, tap=lambda name, h: states.__setitem__(name, h)) | |
| for node in lens.nodes(): | |
| logits = lens.logits(model, states[node][0, -1], node) # unembed(l2norm(J @ h)) | |
| print(node_key(node), [tok.decode([t]) for t in logits.topk(5).indices.tolist()]) | |
| ``` | |
| A lens is fitted for one loop count; run the model with `n_loops = lens.k`. | |
| ## The model, and what a "layer" is | |
| Dory's checkpoint counts sublayers: 72 of them, alternating attention and MLP. Sublayers 0 to 15 are the input | |
| block (run once), 16 to 55 the recurrent block (run once per loop, weights shared), 56 to 71 the output block (run | |
| once). With `x` the input block's output and `N` the L2 norm, loop `t` computes `u_t = N(s_{t-1} + x)` and | |
| `s_t = N(F(u_t))`, starting from `s_0 = x`. Every state is a unit vector. | |
| Lens source nodes, in forward order. Every sublayer output has exactly one node. | |
| | Node | State | | |
| |---|---| | |
| | `emb` | token embedding | | |
| | `in.00` ... `in.14` | after input sublayers 0 to 14 | | |
| | `x` | after sublayer 15, the input block's output | | |
| | `u.t` | input to loop `t` | | |
| | `rec.t.16` ... `rec.t.54` | after recurrent sublayers 16 to 54 in loop `t` | | |
| | `s.t` | after sublayer 55 in loop `t`, the loop's output | | |
| | `out.56` ... `out.70` | after output sublayers 56 to 70 | | |
| The target is the final state (after sublayer 71), which the LM head reads. In `model.forward` the tap names are | |
| tuples, for example `("rec", 2, 41)`; `node_key` turns them into the strings above. | |
| ## How the lens was fitted | |
| The estimator of Gurnee et al. 2026 ("Verbalizable Representations Form a Global Workspace in Language Models"): | |
| for each source node, the Jacobian of the final state with respect to the node's state, summed over target | |
| positions at or after the source position, averaged over source positions and then over prompts. One backward pass | |
| per output dimension (2,560 per prompt, batched). | |
| - 50 prompts: the first 128 tokens of 50 WikiText-103 paragraphs (validation split, seed 0). No BOS token; the | |
| tokenizer adds none. | |
| - The first 16 source positions are left out of the average. | |
| - float32 weights with TF32 matrix multiplies. 39 seconds per prompt at 2 loops on one H200. | |
| - Each lens file also holds the mean state of every node over the fitting positions (`lens.mean_state`), for | |
| centring experiments. | |
| ## Checks | |
| - **Estimator.** On the real model, the autograd Jacobian sums match central finite differences at 10 nodes and | |
| all three position ranges (cosine 0.9998 or higher, norm ratio within 1%). TF32 against exact float32 on the same | |
| rows: cosine at least 0.9999, relative error at most 1.4%. | |
| - **Forward pass.** All 507 checkpoint tensors load. Greedy text is coherent at 1, 2, 4 and 8 loops; loss on | |
| ordinary prose is about 2.0 nats per token (7.8 with the tokens shuffled); the output block applied to a loop-end | |
| state reproduces the model run with that many loops exactly. The code was read step by step against the authors' | |
| transformers pull request (50099) and matches it. | |
| - **Not done.** No number-for-number comparison with the authors' implementation. The sliding-window attention | |
| (1,024 tokens) is exercised on a long document but not validated. | |
| ## Which lens to use | |
| Median split-half cosine by block, for each fitted lens (1 means the lens fitted on the 25 even-numbered prompts | |
| agrees with the one fitted on the 25 odd-numbered prompts): | |
| | File | Loops | Matrices | Size | Input block | Loop 1 | Loop 2 | Loop 3 | Loop 4 | Output block | | |
| |---|---|---|---|---|---|---|---|---|---| | |
| | `jlens_k1.safetensors` | 1 | 73 | 1.9 GB | 0.80 | 0.97 | | | | 1.00 | | |
| | `jlens_k2.safetensors` | 2 | 114 | 3.0 GB | 0.52 | 0.29 | 0.97 | | | 1.00 | | |
| | `jlens_k3.safetensors` | 3 | 155 | 4.1 GB | 0.03 | 0.01 | 0.65 | 0.97 | | 1.00 | | |
| | `jlens_k4.safetensors` | 4 | 196 | 5.1 GB | 0.09 | -0.00 | 0.12 | 0.75 | 0.97 | 1.00 | | |
| - **A lens aimed at the final state reaches back about one loop.** Each lens is stable for its last loop and the | |
| output block, and noise before that. | |
| - **So read loop `t` through the lens fitted at `t` loops.** The states of loops 1 to `t` in a longer run are the | |
| same as in a `t`-loop run, and Dory is trained so that any loop end can go straight to the output block. On 17 | |
| hand-written two-hop prompts (4-loop run, punctuation-like tokens dropped), the intermediate reached the top 10 | |
| at some node of loops 1 / 2 / 3 / 4 on 41 / 47 / 47 / 47% of prompts read this way, against 0 / 0 / 24 / 47% | |
| through the 4-loop lens alone. Exploratory: small hand-made set, best rank over 41 nodes per loop. | |
| - **The per-loop lenses are almost the same matrix.** At the same sublayer, the loop-`t` matrix of the `t`-loop | |
| lens has cosine 0.995 or more with the loop-1 matrix of the 1-loop lens (0.98 at the first recurrent sublayer), | |
| and the output-block matrices agree to 1.000. In practice `jlens_k1.safetensors` alone serves every loop: use its | |
| `rec.1.i` matrix for `rec.t.i`, its `s.1` for `s.t`, and its `out.i` for the output block. | |
| - **The states are almost the same too.** At the same sublayer, consecutive loops' states have cosine 0.95 to 0.99 | |
| between loops 1 and 2 and 0.999 after that (3 prompts), so the readouts of different loops are nearly identical. | |
| Whatever a later loop changes is in a very small part of the state that a readout of the whole state does not | |
| isolate. | |
| ## What the lens is like (2 loops, 50 prompts) | |
| Numbers are from `fit_records/k2_n50_skip16_s0/`; held-out figures are from 8 paragraphs not used in the fit. | |
| | Block | Nodes | Split-half cosine (median) | Top-1 agreement with the model, J-lens | Logit lens | Punctuation-like share of top 10 | | |
| |---|---|---|---|---|---| | |
| | Input block | 17 | 0.52 | 0.00 | 0.00 | 0.66 | | |
| | Loop 1 | 41 | 0.29 | 0.01 | 0.00 | 0.30 | | |
| | Loop 2 | 41 | 0.97 | 0.04 | 0.00 | 0.21 | | |
| | Output block | 15 | 1.00 | 0.41 | 0.29 | 0.10 | | |
| "Split-half cosine" compares the lens fitted on the 25 even-numbered prompts with the one fitted on the 25 | |
| odd-numbered prompts; 1 means the two agree. | |
| - **The input block and loop 1 are not converged.** Their matrices are mostly prompt-to-prompt noise at 50 prompts | |
| (split-half 0.02 to 0.6; it passes 0.8 only from `rec.1.51`). Treat readouts there as unreliable. Loop 2 and the | |
| output block are stable. | |
| - **Readouts are dominated by punctuation and formatting tokens** until the output block. On four two-hop prompts | |
| the unspoken intermediate never reached the top 10 before the output block (best ranks in loop 2: 30 to 373). The | |
| model itself did not answer two of the four directly, so this is not a clean test. | |
| - **The logit lens fails at loop boundaries**: `s.t` and `u.t` are close to one fixed vector, and the unembedding | |
| reads the same few tokens whatever the prompt. | |
| - **Early positions.** No sign of a first-four-tokens problem. Every state in Dory is a unit vector, so there are no | |
| high-norm tokens. Per-position Jacobians are about 3.7 times larger at position 0 and about 2 times larger at | |
| positions 1 to 15 than later (partly because earlier positions have more target positions to sum over). Leaving | |
| out 0, 4 or 16 positions gives nearly the same loop-2 and output-block matrices (cosine 0.99 or higher) and | |
| moderately different input-block ones (0.92). | |
| ## R-lens and J++ (1 and 2 loops, 64 prompts) | |
| Recall@10 of the labelled intermediate on a held-out half of the Gurnee et al. readout sets (multihop, | |
| multilingual, typo, association, poetry; 181 test items), at any of seven nodes equally spaced over the unrolled | |
| depth, macro over tasks, punctuation-like tokens dropped. Brackets: 95% bootstrap interval over test items. | |
| | Lens | 1 loop | 2 loops | | |
| |---|---|---| | |
| | Logit lens | 0.244 [0.18, 0.31] | 0.189 [0.14, 0.25] | | |
| | Plain J-lens | 0.319 [0.26, 0.39] | 0.268 [0.22, 0.33] | | |
| | R-lens | 0.396 [0.33, 0.46] | 0.321 [0.26, 0.39] | | |
| | J++ | 0.465 [0.40, 0.53] | 0.423 [0.36, 0.49] | | |
| | J++, weights learned on shuffled labels | 0.238 | 0.225 | | |
| - J++ minus R-lens: +0.069 [+0.03, +0.12] at 1 loop, +0.102 [+0.02, +0.18] at 2 loops. | |
| - With the LRP rules the lens is stable in every block (split-half cosine 0.99), including loop 1 of a 2-loop run. | |
| - Limits: the J++ weights were learned on the other half of the same five tasks; no swap or ablation test has | |
| been run; multihop has 10 or 11 test items because Dory answers about a quarter of them; one seed, one router, | |
| one split. | |
| - LRP choices specific to Dory (not covered by the R-lens or J++ posts): every L2 norm's denominator is constant | |
| except the query and key norms, and the attention output gate is treated as a constant. | |
| - Records: `fit_records/jpp_k1_n64_e8_s0_r2/` and `fit_records/jpp_k2_n64_e8_s0/`. Lens files: `lenses/jpp_k<k>.safetensors` and `lenses/rlens_k<k>.safetensors`. | |
| ## Not here yet | |
| A centred readout, causal tests of the lens directions, and a multi-token readout (Dory has no multi-token | |
| prediction head, so the chain lens does not apply directly). | |