| --- |
| tags: |
| - interpretability |
| - j-space |
| - mixture-of-experts |
| license: apache-2.0 |
| --- |
| |
| # Inkling J-space lens (n=250, all-layer) |
|
|
|
|
| A **Jacobian lens (J-space) fit of [thinkingmachines/Inkling](https://huggingface.co/thinkingmachines/Inkling)** (~950B-param MoE: 66 layers, |
| d_model 6144, 256 routed experts/layer + 2 shared, hybrid SWA-512/global attention, sconv), |
| fitted from the official NVFP4 checkpoint ([thinkingmachines/Inkling-NVFP4](https://huggingface.co/thinkingmachines/Inkling-NVFP4)). |
| |
| **J_ℓ = E[∂h_target,t′ / ∂h_ℓ,t]** — the expected Jacobian of the final residual stream with |
| respect to layer ℓ's residual stream, averaged over token positions and prompts. |
| |
|  |
| |
| |
| ## Fit configuration |
| |
| | | | |
| |---|---| |
| | source layers | **all 65** (0–64), target = layer 65 | |
| | prompts | **250** wikitext-103 (train, max_chars 2000), seq_len 128, 2-way prompt-sharded | |
| | weights | NVFP4 (block-16 e4m3 scales) resident, bit-exact fused dequant-autograd (bf16 compute, fp32 accumulation) | |
| | dim_batch | 64 (96 backward passes/prompt) | |
| | rate | ~526 s/prompt/node on 8×H200, 18.6 h/shard, 2 nodes | |
| | merge | exact n-weighted: J = (n₀·J₀ + n₁·J₁)/(n₀+n₁), verified ≤3.1e-5 | |
|
|
| ## Files |
|
|
| - `inkling_jacobian_lens_n250.pt` — merged lens (`jlens` JacobianLens payload: per-layer fp32 |
| `J[6144,6144]` means for layers 0–64, target 65, n=250) |
| - `shards/shard0of2.pt`, `shards/shard1of2.pt` — the two disjoint-prompt half-lenses |
| (125 prompts each; used for the split-half robustness checks — same-layer CKA median 0.997) |
| - `merge_weights.json`, `convergence_shard{0,1}.csv` — provenance |
| - `cka/` — the recombined CKA space: `inkling_cka.npz` (65×65 CKA-vs-ρ matrix), |
| `inkling_cka_blocks.json` (KTS-DP segmentation), `inkling_cka_heatmap.png`, `summary.json` |
