| --- |
| tags: |
| - interpretability |
| - jacobian-lens |
| - nvfp4 |
| license: apache-2.0 |
| --- |
| |
| # Laguna-XS-2.1 J-lens: bf16 vs NVFP4 A/B |
|
|
| Two Jacobian lenses of [poolside/Laguna-XS-2.1](https://huggingface.co/poolside/Laguna-XS-2.1) (~33B fine-grained MoE, 40 layers, d=2048, |
| 256 experts/8 active) fitted on **byte-identical prompts** (200 × wikitext-103, seq 128, |
| all 39 source layers, target 39) — one from the bf16 checkpoint, one from the official |
| NVFP4 twin. The twins' non-expert tensors are bit-identical (verified: lm_head, final |
| norm, embeddings, attention, router), so every lens difference is attributable to |
| **NVFP4 quantization of expert weights** (plus one ~0.6% export-time perturbation of the |
| 3 layer-0 dense-MLP tensors, documented in the report). |
| |
| |
|  |
| |
| ## Files |
| |
| - `laguna_bf16_jacobian_lens.pt`, `laguna_nvfp4_jacobian_lens.pt` — merged n=200 lenses (fp16 payload; fp16 rounding is irrelevant at 1−CKA ≈ 2e-9) |
| - `shards_{bf16,nvfp4}/` — 8 per-GPU shard lenses per arm (fp32, 25 prompts each; shard i = same prompts in both arms) |
| - `convergence_{bf16,nvfp4}/` — per-shard convergence CSVs |
| - `results/` — `ab_summary.json`, per-analysis npz, figures, `REPORT.md` |
|
|
| Probe: P = W_U[ids]·γ (plain-γ RMSNorm, no softcap), ids = 4096 frozen seed-0 over |
| vocab 100352; identical across arms. Fit code: `open-jlens-data` repo |
| (`code/moe/fit_moe.py` laguna dispatch, `nvfp4_experts.py`, `analyze_laguna_ab.py`). |
| Fitted + analyzed 2026-07-17. Context: this A/B calibrates the NVFP4 caveat of the |
| Inkling ~950B lens ([PrimeIntellect/inkling-jlens](https://huggingface.co/PrimeIntellect/inkling-jlens)) — same modelopt recipe family; |
| transfer caveat: Laguna experts are 512-dim vs Inkling 3072-dim. |
| |
| ## xs2/ — Laguna-XS.2 version-drift extension (E9) |
| |
| `xs2/` holds the same fit for **poolside/Laguna-XS.2** (bf16, identical 200 prompts/probe/tokenizer) |
| plus the version comparison vs XS-2.1. Verdict: **version drift is a regime change** — same-layer |
| CKA median 0.638 = 43× the NVFP4 effect and 54× prompt noise — while the depth architecture |
| survives (both versions k=2 at layer 27/28, cross-matrix depth correlation 0.990). The versions |
| share no token basis (same-token unembedding rows near-orthogonal), so each is read through its |
| own probe. Perturbation ladder on identical prompts (1−CKA): noise 0.007 < NVFP4 0.008 ≪ version 0.362. |
| |
