docs/PROTOCOL.md

Full precision Mamba2-8B Resurface control

Declared 2026-09-28 before fitting this repository's adapter. The purpose is to measure the effect of the same readout adapter and training budget on NVIDIA's uncompressed pure Mamba2-8B. The E8/W5 experiment is a historical comparator, not a training dependency. All metrics are reported at the actually measured precision and protocol, without assuming equality to native Megatron BF16.

Frozen model and intervention

Load nvidia/mamba2-8b-3t-4k revision b915550c63ba9359f88f44d1f6a600d85af27302, checkpoint SHA256 47c2766f6aad89d73beafbeaecb334aab902d7370906d081764a90bb7a8bbbcb and tokenizer SHA256 5862e2f71caf762bc9845662be5fec2867deb58d874568235a02a36c5111cd09. Cast the 507 original BF16 tensors to FP16 in the native state-spaces Mamba2 runtime; freeze all 8,236,999,680 base parameters. Verify strict source tensor mapping, model geometry and no shared embedding/output storage.

Use the same post-D adapter as the compressed experiment at all 56 native gated RMSNorm inputs: y + sigmoid(w dot u+b) * g*(V@y) across 128 heads. Its 224 tensors contain 1,154,104 FP32 trainable master parameters, exported as FP16 for inference. Initialize V=0, g=1, w=0, b=-4; use soft sigmoid at train and eval, no task switch, EMA, or new recurrent cache. The adapter is external to the frozen base; check its identities and absence of base grads.

Training

Reuse the same exact data generator contract and seeds as the compressed adapter: 1,536 TRAIN numeric bindings over three templates and N=16/64, disjoint six-digit key/value intervals; full 256K CE on every answer suffix token. Use seed 2026092803 and torch.randperm(1536) once for example order. Each step pairs one MK example with a 512-token segment from the already prepared 448 WikiText-2 TRAIN windows. Their historical manifest and file SHA256 are pinned by the train command. Heldout and validation windows are not included in training. Reusing this fixed TRAIN text gives the control the same training exposures as the compressed arm.

Use a second, independently loaded uncompressed FP16 model as the frozen prose teacher. This is the direct counterpart to the compressed experiment's teacher, which was its own unadapted compressed base. The fixed per-step loss is MK answer CE + 0.5 prose CE + 0.5 KL(own unadapted teacher || student) + 3 C, where C is the same prose-only router closure, with budget 0.006 and excess coefficient 10. Temperature 1, 511 prose targets per step, 64-token staged head chunks, and one optimizer update after both task gradients.

AdamW FP32 masters: V/g LR 1e-4, w/b LR 3e-4, betas (.9,.999), eps 1e-8, weight decay 0, clip norm 1. Multiplicative schedule 0.1+0.9*(1+cos(pi*j/1535))/2 at successful index j. Native FP16 forward, block checkpointing, gradient scale 1024 with growth interval 2000. Overflow retries the exact same pair without optimizer/master update, at most 8 retries and 1,544 total attempts. Run 1,536 successful updates; the final step is the only candidate. Small GPU smokes are discarded before formal training.

Evaluation and interpretation

After training, restore the actual serialized FP16 adapter onto the same uncompressed base. Evaluate baseline, adapter enabled and restored baseline in a single process, with identical tokenizer, MK prompts and full WikiText-2 validation windows. For MK use full 256K greedy generation up to 12 tokens, first standalone six-digit number, normal and target-removed controls, native prefill plus recurrent decode with fresh FP16 cache. For PPL use all 130 nonoverlapping reset windows and all 264,764 next-token targets. Record all raw scores, differences and file hashes; no early selection or task switch.

The earlier compressed arm's independent CONFIRM set has now been observed and uses the same three template families. Running this source control on it gives a matched historical comparison, not a newly untouched holdout. The WikiText-2 validation text also informed earlier development. We will not promote a source+adapter result to a general recall conclusion without new templates, longer distances and a truly untouched corpus.

Report all four observed arms explicitly: source FP16, source+adapter, compressed/readapted, compressed/readapted+adapter. The source and compressed teachers differ by construction. Report PPL and recall together; any cross-arm claim remains limited by shared test protocols and the historical numerical replay discrepancy in the compressed project.