---
tags:
- interpretability
- j-space
- mixture-of-experts
license: apache-2.0
---

# Inkling J-space lens (n=250, all-layer)


A **Jacobian lens (J-space) fit of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling)** (~950B-param MoE: 66 layers,
d_model 6144, 256 routed experts/layer + 2 shared, hybrid SWA-512/global attention, sconv),
fitted from the official NVFP4 checkpoint ([thinkingmachines/Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4)).

**J_ℓ = E[∂h_target,t′ / ∂h_ℓ,t]** — the expected Jacobian of the final residual stream with
respect to layer ℓ's residual stream, averaged over token positions and prompts.

![Capture d’écran 2026-07-17 à 19.32.08](https://cdn-uploads.huggingface.co/production/uploads/651e96991b97c9f33d26bde6/CIMENSTykyE_r4vZ7mAMi.png)


## Fit configuration

| | |
|---|---|
| source layers | **all 65** (0–64), target = layer 65 |
| prompts | **250** wikitext-103 (train, max_chars 2000), seq_len 128, 2-way prompt-sharded |
| weights | NVFP4 (block-16 e4m3 scales) resident, bit-exact fused dequant-autograd (bf16 compute, fp32 accumulation) |
| dim_batch | 64 (96 backward passes/prompt) |
| rate | ~526 s/prompt/node on 8×H200, 18.6 h/shard, 2 nodes |
| merge | exact n-weighted: J = (n₀·J₀ + n₁·J₁)/(n₀+n₁), verified ≤3.1e-5 |

## Files

- `inkling_jacobian_lens_n250.pt` — merged lens (`jlens` JacobianLens payload: per-layer fp32
  `J[6144,6144]` means for layers 0–64, target 65, n=250)
- `shards/shard0of2.pt`, `shards/shard1of2.pt` — the two disjoint-prompt half-lenses
  (125 prompts each; used for the split-half robustness checks — same-layer CKA median 0.997)
- `merge_weights.json`, `convergence_shard{0,1}.csv` — provenance
- `cka/` — the recombined CKA space: `inkling_cka.npz` (65×65 CKA-vs-ρ matrix),
  `inkling_cka_blocks.json` (KTS-DP segmentation), `inkling_cka_heatmap.png`, `summary.json`