File size: 1,949 Bytes
e3c97b1 4594d6f e3c97b1 c9d03e8 f1373de e3c97b1 f1373de e3c97b1 c9d03e8 e3c97b1 4594d6f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 | ---
tags:
- interpretability
- j-space
- mixture-of-experts
license: apache-2.0
---
# Inkling J-space lens (n=250, all-layer)
A **Jacobian lens (J-space) fit of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling)** (~950B-param MoE: 66 layers,
d_model 6144, 256 routed experts/layer + 2 shared, hybrid SWA-512/global attention, sconv),
fitted from the official NVFP4 checkpoint ([thinkingmachines/Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4)).
**J_ℓ = E[∂h_target,t′ / ∂h_ℓ,t]** — the expected Jacobian of the final residual stream with
respect to layer ℓ's residual stream, averaged over token positions and prompts.
## Fit configuration
| | |
|---|---|
| source layers | **all 65** (0–64), target = layer 65 |
| prompts | **250** wikitext-103 (train, max_chars 2000), seq_len 128, 2-way prompt-sharded |
| weights | NVFP4 (block-16 e4m3 scales) resident, bit-exact fused dequant-autograd (bf16 compute, fp32 accumulation) |
| dim_batch | 64 (96 backward passes/prompt) |
| rate | ~526 s/prompt/node on 8×H200, 18.6 h/shard, 2 nodes |
| merge | exact n-weighted: J = (n₀·J₀ + n₁·J₁)/(n₀+n₁), verified ≤3.1e-5 |
## Files
- `inkling_jacobian_lens_n250.pt` — merged lens (`jlens` JacobianLens payload: per-layer fp32
`J[6144,6144]` means for layers 0–64, target 65, n=250)
- `shards/shard0of2.pt`, `shards/shard1of2.pt` — the two disjoint-prompt half-lenses
(125 prompts each; used for the split-half robustness checks — same-layer CKA median 0.997)
- `merge_weights.json`, `convergence_shard{0,1}.csv` — provenance
- `cka/` — the recombined CKA space: `inkling_cka.npz` (65×65 CKA-vs-ρ matrix),
`inkling_cka_blocks.json` (KTS-DP segmentation), `inkling_cka_heatmap.png`, `summary.json` |