Blame for code/docs/MEASUREMENTS.md

# Measurements: first GPU evidence pass (G0 control plane on NCCL, P0-M MoE dispatch)

Raw numbers from the first real-GPU run of the G0 work, with what each one decides in `docs/ROADMAP.md`. Nothing here is an estimate: every figure is copied from a file under `.tmp/gpu_smoke/` (named per section). Scope is deliberately small: tiny models, 2 GPUs, one single-GPU benchmark. **Nothing here is a 4-rank, 7B, or multi-minute result**, and sections "Not measured" lists what that leaves open.

## Environment

| Item | Value |
|---|---|
| Date / window | 2026-10-12, 17:45 to 17:57 IST (host `pl-gpt-machine`) |
| GPUs used | physical GPU 1 (`GPU-bdbde387-01e3-14d4-9a9a-5d40e8cf3427`, PCI 00:1C.0) and GPU 0 (`GPU-c569cb57-edc9-3dd1-5e5e-6822ebcb56f0`, PCI 00:1B.0). Sections (a), (b): `CUDA_VISIBLE_DEVICES=1,0` (rank 0 -> GPU 1, rank 1 -> GPU 0). Sections (c), (d): GPU 1 alone. |
| GPU | NVIDIA A10G, sm_86, 23028 MiB, SM clock 1710 MHz (max 1710), 35-42 C, 58-62 W idle, 133 W during the benchmark. PCIe gen4 x8; all four GPUs `PHB` to each other (no NVLink). |
| Driver / CUDA / torch / triton / NCCL | 580.65.06 / 12.8 / 2.11.0+cu128 / 3.6.0 / 2.28.9+cuda12.9 |
| Python | `/apps/development/rlhf/align/.venv/bin/python` (3.12), `PYTHONPATH=/apps/development/rlhf/align` (the package is not installed in the venv) |
| Foreign load on the GPUs | one idle `nmo_trace_server.py` (circuit-tracer UI, not ours) holds 12045 / 10853 / 12045 / 12045 MiB on GPUs 0-3 at 0% utilisation for the whole window. Only ~11 GiB per card was free; nothing here needed more than 4.4 GiB. Its presence did not change utilisation (0% before and after) but a benchmark taken next to an idle resident context is not a clean-box number. |
| Raw files | `.tmp/gpu_smoke/{bench_c.json,bench_c2.json,diag_d.json,runs/*/}`; scripts `bench_c.py`, `bench_c2.py`, `diag_d.py`, `worker.py`, `driver.py`, `compare.py`, `preflight.sh` (all scratch, not committed) |

### Preflight (all passed; re-run before every GPU command, last values shown)

```
MemAvailable_GB=121  load1=1.06  diskq(nvme2n1)=0  D-state=0  date=2026-10-12T17:47:41+05:30
GPU  used/total MiB  util
0    12045 / 23028   0 %
1    10853 / 23028   0 %
2    12045 / 23028   0 %
3    12045 / 23028   0 %
PICK = GPUs 1,0   (least memory used; each >= 4 GB free, < 50% util)
```
Thresholds: MemAvailable >= 45 GB, load < 96, disk queue < 64, D-state <= 5. Heavy-job lock (`.align_heavy_job.lock`) was free before and was taken by rank 0 of each torchrun via `apply_resource_guards(heavy=True)`. After the last run: the only GPU processes were the four foreign server pids, same memory as before.

## (a) FSDP2 + adamw8bit + per-rank optimizer state: SIGTERM one rank, resume, compare

**Setup.** `worker.py` under `torchrun --standalone --nproc_per_node=2`, NCCL. Tiny dense align Mistral (4 layers, d=256, 8 q / 4 kv heads, vocab 1024, bf16, 2,885,888 parameters, 39 tensors), production wrapping (`apply_activation_checkpointing` then `apply_fsdp`: per-block FSDP2 + root, bf16 params, fp32 reduce), `optimizer.name=adamw8bit` (`max_grad_norm=1.0`, 1.4M local params, block-wise 8-bit moments on device), `train.per_rank_optimizer_state=true`, `grad_accum=2`, 20 micro-steps (10 optimizer steps), `save_every_micro_steps=10`, `tripwires.armed=("grad_norm",)` (as `configs/sft.toml`), `ALIGN_PG_TIMEOUT_S=120`. Each rank trains on its own rows. Peak device memory per rank: 0.034 GiB.

**Interrupt.** `driver.py` polls `history.jsonl` and sends a real external `SIGTERM` to **rank 1's pid only** once micro-step 15 is logged (a 0.4 s sleep per micro-step makes the timing reliable). It was sent at t = 13.08 s after launch.

| Run | What | Result |
|---|---|---|
| `a_full1`, `a_full2` | uninterrupted, 20 micro-steps, same seed | final params **bit-identical** (0 of 2,885,888 elements differ). Run-to-run nondeterminism of this setup on 2 GPUs is zero. |
| `a_int` | SIGTERM to rank 1 at logged micro 15 | **both** ranks logged `coordinated stop (signal) at optimizer step 8 (micro 16)` (the next optimizer-step boundary), both returned `stop_exit_code` = 143 (128+15), checkpoints `micro_00000010` and `micro_00000016`; `checkpoint_problem()` returns `None` (complete) for both; each holds `model.safetensors`, `resume.pt`, `optimizer_rank0.pt`, `optimizer_rank1.pt`, tokenizer files. No `HALT.json` (a signal stop is not a verdict). `torchrun` itself exited 1, as the integration report measured. |
| `a_res16` | resume `latest` (= the stop checkpoint, micro 16), finish to 20 | `resumed_micro_step` 16, finished at micro 20 / optimizer step 10, exit 0. |
| `a_res10` | resume the explicit cadence checkpoint `micro_00000010`, finish to 20 | `resumed_micro_step` 10, finished at 20 / 10, exit 0. |

**Final-parameter comparison against the uninterrupted run (`compare.py`, bf16 params gathered with `full_state_dict()`):**

| Comparison | max abs diff | differing elements | verdict |
|---|---|---|---|
| `a_full1` vs `a_full2` (run-to-run) | 0.0 | 0 / 2,885,888 | bit-exact |
| `a_full1` vs `a_res16` (resume from the SIGTERM stop checkpoint) | **0.0** | **0 / 2,885,888** | **bit-exact** |
| `a_full1` vs `a_res10` (resume from the cadence checkpoint) | **0.0** | **0 / 2,885,888** | **bit-exact** |
| sensitivity: `a_full1`@20 vs `a_int`@16 (different training stage) | 0.0273 | 2,141,203 | differs, so the comparison can see change |
| negative control: resume from 16 with `optimizer_rank0.pt` and `optimizer_rank1.pt` swapped | 0.0273 | 2,333,055 | differs, so a wrong optimizer shard is detected, i.e. the bit-exact result depends on the per-rank files being read correctly |

Loss, lr and grad-norm in `history.jsonl` also match the uninterrupted run at every micro-step the interrupted and resumed runs share: `a_int` 16 of 16, `a_res16` 20 of 20, and the resumed portion (micros 11-20) of `a_res10`. Bit-exactness here is explained, not lucky: with 2 ranks an NCCL reduce-scatter sums two values (commutative, so order-free), params are bf16 with no master copy so the checkpoint round trip is lossless, 8-bit moments are saved in their quantised form, and RNG/stream/scheduler state are restored. A 4-rank NCCL ring sums in a fixed ring order too, but that is **not** shown here.

## (b) NaN on one rank: skip cap HALTs both ranks, no NCCL hang

Same setup, `train.max_consecutive_skipped_steps=3`. The adapter multiplies the loss by NaN on **one rank only**. (The injection index is `trainer.micro_step + 1`, so `--nan-from 5` first hits micro_step 4. The table uses the real micro_step.)

| Run | Injection | Result |
|---|---|---|
| `b_halt` | rank 1, every micro-step from 4 | both ranks raised `NonFiniteHaltError` at **micro_step 10** (optimizer step 1), identical message ("4 consecutive bad optimizer steps exceed train.max_consecutive_skipped_steps=3 (last cause: non-finite loss and skipped step)"), `skip_stats = {consecutive 4, bad_total 4, boundaries 5}`, `optimizer_skipped_steps` 4 on both, exit code 81 on both, `HALT.json` `{"code": 81}` written. Worker-side time from the start of `main` to the raise was 1.24 s (rank 0) and 1.39 s (rank 1); the whole launch took 7.8 s. No NCCL error, watchdog or timeout line in the log (the only `NCCL` matches are the version banner and a deprecation warning, see Findings). No checkpoint existed yet (first cadence save is micro 10), so none was written. |
| `b_halt_rank0` | rank **0**, every micro-step from 4 | identical outcome with the roles swapped: both ranks exit 81 at micro_step 10, 1.2 s. |
| `b_transient` | rank 1, **one** micro-step (real micro_step 6) | not a halt. Both ranks `skip_stats = {consecutive 0, bad_total 1, boundaries 10}`, `optimizer_skipped_steps` 1, 9 of 10 optimizer steps applied, finished at micro 20, exit 0, all parameters finite. |

The grad gate's skip is global (NaN in one rank's gradient reaches both shards through the reduce-scatter, and the global-norm is non-finite on both), and the non-finite-loss flag is MAX-reduced with the stop flags in the one boundary collective, so both ranks always take the same branch.

## (c) MoE expert dispatch: `loop` vs `grouped_mm`, one A10G (GPU 1)

**Shape.** E=64 experts, top-8, d_model 2048, expert FFN 1024 (SwiGLU: `gate_up` [64, 2048, 2048], `down` [64, 2048, 1024]), bf16, `align.model.moe.moe_experts_forward`. Median of 10 after 3 warmups, CUDA events, includes gather, both expert matmuls, SiLU*up, routing-weight scale and `index_add_` combine; the router itself is excluded from this table (timed in the breakdown below). "Balanced" = the real router on random activations; "skewed" = a Zipf(1.0) expert popularity, k distinct experts per token (max expert load 7.1x the mean, no empty expert). Backward uses a gradient equal to the output; `weights` and `x` require grad. FLOPs per layer forward = 2*T*k*(d*2I + I*d) = 412 GFLOP at T=4096, 1649 GFLOP at T=16384; backward counted as 2x forward.

| T | routing | loop fwd ms | loop fwd+bwd ms | grouped fwd ms | grouped fwd+bwd ms | fwd+bwd speedup | loop / grouped peak GiB | fwd+bwd TFLOP/s loop / grouped |
|---|---|---|---|---|---|---|---|---|
| 4096 | balanced | 13.132 | 39.684 | 12.388 | 35.325 | 1.123x | 2.17 / 1.92 | 31.2 / 35.0 |
| 4096 | skewed | 14.557 | 42.345 | 13.732 | 37.518 | 1.129x | 2.17 / 1.92 | 29.2 / 33.0 |
| 16384 | balanced | 49.139 | 136.851 | 46.440 | 128.696 | 1.063x | 4.40 / 4.40 | 36.2 / 38.4 |
| 16384 | skewed | 47.846 | 135.295 | 45.353 | 127.205 | 1.064x | 4.40 / 4.40 | 36.6 / 38.9 |

(min/max over the 10 repeats are within 1% of the median in `bench_c.json`.)

**Is the fallback taken? Yes, confirmed three independent ways.**
1. `TORCH_CPP_LOG_LEVEL=INFO` on a 2-call probe prints, on **every** call: `[I... GroupedMMUtils.h:128] fallback path for `torch._grouped_mm`, performance may not be optimal` (`probe_log.err`). The same string is in `libtorch_cuda.so` next to a `_grouped_mm_fallback` symbol.
2. The profiler on one `F.grouped_mm(x_sorted, gate_up^T, offs)` call (T=4096 balanced) shows **64** `aten::mm` and **64** `aten::item` CPU ops and **93** CUDA kernels: 64 cuBLAS GEMMs (36x `ampere_bf16_s16816gemm_bf16_128x128_ldg8_f2f_stages_32x5_tn`, 28x `ampere_bf16_s1688gemm_bf16_128x128_ldg8_f2f_stages_32x1_tn`), 28 memsets, and **1 `Memcpy DtoH`** (the `offs` read-back that forces a host sync). No CUTLASS grouped kernel launches.
3. Timing: grouped_mm is only 6-13% faster than the loop (above), which is what swapping one loop of 64 cuBLAS calls for another loop of 64 cuBLAS calls looks like.

**Agreement of the two backends on the same inputs** (bf16, T=4096 balanced; the other three rows are the same within noise): output `y` max abs diff 2.4e-3 on an absmax of 0.465 (one bf16 ulp at that magnitude, relative 5.3e-3); `dx` 3.9e-3 on 0.652 (6.0e-3); **`dgate_up`, `ddown`, `dweights` max abs diff exactly 0.0** (bitwise identical weight and routing-weight gradients). Hypothesis, not measured further: the forward 1-ulp difference is the cuBLAS kernel choice (the `s1688` vs `s16816` split above) plus the nondeterministic `index_add_` order.

**Where the time goes** (`bench_c2.json`, loop backend forward, balanced routing, ms):

| Component | T=4096 | T=16384 |
|---|---|---|
| `route_top_k` (fp32 router + softmax + top-k) | 0.273 | 1.120 |
| `build_dispatch_plan` (stable sort + scatter_add counts) | 0.210 | 0.209 |
| `counts.tolist()` host sync (idle stream) | 0.032 | 0.032 |
| gather `index_select` | 0.552 | 2.238 |
| 64x `gate_up` linears | 5.536 | 20.806 |
| SiLU * up | 0.726 | 2.817 |
| 64x `down` linears | 3.056 | 10.962 |
| scale + `index_add_` combine | 2.317 | 9.374 |
| **full loop forward (measured)** | **13.162** | **49.225** |
| sum of the seven non-router components | 12.429 | 46.438 |
| GEMM share of the loop forward | 65.3% | 64.5% |
| GEMM throughput inside the layer | 48.0 TFLOP/s | 51.9 TFLOP/s |
| whole layer forward throughput | 31.3 TFLOP/s | 33.5 TFLOP/s |

**Dense-GEMM ceiling on this card** (cuBLAS bf16, same GPU, same clocks): per-expert shapes run as 64 back-to-back `F.linear`: 512 rows/expert: gate_up 48.8, down 43.8 TFLOP/s; 2048 rows/expert: gate_up 52.9, down 50.2 TFLOP/s; one large `8192x4096 @ 4096x4096`: **68.0 TFLOP/s**.

## (d) Tiny OLMoE on CUDA, both dispatch backends

`diag_d.py` (GPU 1): OLMoE architecture, 2 layers, d=128, expert width 64, 16 experts top-4, 4 heads, vocab 512, sdpa attention, identical weights loaded into a `loop` and a `grouped_mm` model, batch 4x32, CE loss, full backward.

| dtype | logits max abs diff (absmax) | loss loop / grouped | router counts identical | worst grad rel diff | params with all-zero grad | finite | pass |
|---|---|---|---|---|---|---|---|
| fp32 | 1.9e-7 (0.908) | 6.259638 / 6.259639 | yes (every layer) | 6.7e-7 (`layers.1.attn.k_proj.weight`) | 0 | yes | yes (tol 1e-4) |
| bf16 | 3.9e-3 (0.906) | 6.259564 / 6.259566 | yes (every layer) | 1.0e-2 (`layers.0.mlp.experts.down_proj`, 5.7e-6 on 5.6e-4) | 0 | yes | yes (tol 5e-2) |

`grouped_mm_unavailable_reason("cuda")` returned `None` (the 2-expert probe runs on sm_86: the fallback works, it is just not fast). Every parameter, including router and stacked experts, receives a non-zero gradient on CUDA.

## What each number decides

| Number | Decision it feeds |
|---|---|
| grouped_mm is the fallback on sm_86 (log line, 64 mm + 64 item + 1 DtoH, 93 kernels) | Closes the "expect the slow fallback" half of **P1-11** with evidence. `docs/REFERENCE_GAP_ANALYSIS.md` line for `torch._grouped_mm` ("A10G is firm degraded") is now measured, not inferred. |
| grouped_mm 1.06-1.13x faster than loop (fwd+bwd), same memory, weight grads bit-identical | Either backend is correct on A10G; the choice is worth at most ~6-12%. Keep `loop` the default (oracle, no extra probe/alignment constraint); `grouped_mm` may be used where `grouped_mm_shape_problem` is None. **Not** a reason to build anything. |
| GEMMs are 65% of layer forward and run at 48-52 TFLOP/s in-layer, 44-53 TFLOP/s for the same shapes in isolation, vs 68 TFLOP/s for one large GEMM | **P1-12 (Triton grouped GEMM) fails its own entry condition** ("only if P1-11 shows the loop is the bottleneck"): the loop's GEMM efficiency equals cuBLAS on those shapes, so a Triton kernel's best case is closing 49 -> ~68 TFLOP/s on 65% of the layer, at most ~18% of forward time (an upper bound: cuBLAS itself does not reach 68 on these shapes), with a high bar to beat cuBLAS on sm_86 with a 99 KB shared-memory limit. Recommend **defer P1-12** and spend the effort on the glue below, if anything. |
| Non-GEMM glue = 35% of layer forward: combine (`y*w` + `index_add_`) 17.6% / 19.0%, gather 4.2% / 4.5%, SiLU*up 5.5% / 5.7%, routing + plan 3.7% / 2.7% (T=4096 / 16384) | If a custom kernel is worth writing for MoE on A10G it is a fused gather and a fused scale+combine (+ SiLU*up), not the GEMM. Worth a P1 line item **only after** P1-15 shows MoE layers dominate step time. |
| Host sync in the loop (`counts.tolist()`) costs 0.032 ms on an idle stream | Isolated cost is negligible. Its real cost (draining the launch queue once per layer and blocking CPU run-ahead) is **not** measured by this benchmark; P1-15's end-to-end MFU is the test. Do not remove it on this evidence. |
| Layer fwd+bwd at 4096 tokens = 35-40 ms (loop 39.7), 16384 tokens = 127-137 ms; 31-39 TFLOP/s whole-layer | First anchor for the OLMoE-1B-7B throughput estimate (16 MoE layers, ~0.6 s/micro-batch of expert compute at 4096 tokens/GPU before attention, optimizer and comms), to be replaced by the P1-15 measured MFU. The analysis doc's 25 TFLOP/s assumption is below this single-GPU layer figure, as expected before comms. |
| Peak 2.2 GiB (4096) / 4.4 GiB (16384) for one layer incl. 0.75 GiB expert weights + 0.75 GiB grads | Per-layer transient for the P1-15 memory arithmetic: peak minus the 1.5 GiB of weights and grads is ~0.7 GiB at 4096 tokens and ~2.9 GiB at 16384 (before attention, residual stream and activation checkpointing, which this benchmark does not include). |
| (d): loop and grouped_mm agree on CUDA end to end in fp32 (1e-7 / 7e-7) and bf16 (1 ulp / 1e-2 worst grad), identical routing, no dead parameter | Clears the **CUDA** half of the P5 dispatch parity claim (previously CPU only). Backend choice does not need a model-level gate. |
| (a): resume from a SIGTERM stop checkpoint and from a cadence checkpoint is **bit-exact** vs an uninterrupted run on 2 NCCL ranks, FSDP2 + adamw8bit + per-rank optimizer files; negative control detects wrong shards | Moves **P0-1 / P0-8 / P0-9 / per-rank** from "CPU-verified" to "NCCL 2-rank verified" for the lossless-resume and coordinated-stop claims. Leaves open: 4 ranks, 7B-scale save/restore timing, `kill -9` mid-save. |
| (a): coordinated stop took 1 boundary (micro 15 signal -> micro 16 stop), both ranks, 143 on both | The boundary-latency of a stop is `grad_accum` micro-steps. At accum 8 and ~minutes per micro-step on the 7B job this is the number to budget in the supervisor's SIGTERM grace period; **it needs the 7B step time to turn into seconds**. |
| (b): persistent NaN on either single rank halts both at the same micro-step in ~1.3 s with identical error and exit 81, no NCCL hang; a single transient NaN skips exactly one step and the run continues healthy | Moves **P0-2** from "CPU gloo" to "NCCL 2-rank verified": the skip-cap/non-finite agreement needs no extra collective to avoid the lone-raiser hang. The remaining P0-2 item (`GradNormTripwire` treating a NaN norm as blind) is unchanged. |

## Findings that were not asked for

1. **P0-6 (rank-local logging) is confirmed on real ranks.** `history.jsonl` receives one row per rank per micro-step: 40 rows for 20 micro-steps at world 2, rows from the two ranks carry different per-rank losses, and under the rank-1 NaN injection the file shows a NaN loss row (rank 1) and a finite row (rank 0) for the same micro-step. Nothing interleaved or corrupted a line in these runs. A consumer reading "the last row" or counting rows gets rank-dependent answers. Reduce at log cadence and write from rank 0 only, as P0-6 says.
2. **`TORCH_NCCL_TRACE_BUFFER_SIZE` is deprecated on torch 2.11** (`Warning: Environment variable TORCH_NCCL_TRACE_BUFFER_SIZE is deprecated; use TORCH_FR_BUFFER_SIZE instead`, 4 lines per launch). `configure_nccl_diagnostics` sets the old name for the buffer size (it sets both names only for the dump-file prefix). Whether the old name still enables the recorder was not tested: `nccl_trace/` was created and is empty because no collective timed out. P0-10's open question (does `DUMP_ON_TIMEOUT` need `TORCH_NCCL_ENABLE_MONITORING`) still needs a deliberate hang, not done here.
3. **The guards caught a real launch-environment fault.** The first launch was refused with `writable path '/home/vignesh/.opencode/tmp' resolves to device 64768, not the /apps device`: the Claude session's `TMPDIR` points at the root filesystem. `driver.py` now unsets it so `apply_resource_guards` sets `/apps/.../align/.tmp`. Sections (c) and (d) were single-process scripts that did **not** go through `apply_resource_guards` (no heavy-job lock, inherited `TMPDIR`); they wrote nothing outside `.tmp/gpu_smoke/`.
4. **`torchrun` reaps the survivor.** After rank 1 exited (143) torchrun logged `Sending process ... closing signal SIGTERM` for rank 0 and exited 1. The checkpoint was complete because `save_checkpoint` ends in a barrier, so a peer cannot exit before rank 0 finishes writing; in these runs rank 0 also completed its own post-train dump. Anything a stage script does after `train()` returns on rank 0 only (an eval, a copy) can be cut off by this; the existing scripts return immediately on a coordinated stop, which is the safe shape.
5. The tiny runs do not learn (loss 6.96 to 7.02 over 20 steps on random tokens); they validate the control plane and optimizer plumbing (`trunk_report.moved_fraction` = 1.0, all 39 tensors change between micro 16 and 20), not optimisation quality.

## Not measured / still open

- 4 ranks, a 7B model, FSDP2 with MoE (the OLMoE `RouterStats` through FSDP2 hooks), expert sharding on dim 0, EP all-to-all (P1-13), any run longer than 10 seconds, NCCL behaviour under a real hang, `kill -9` during a save, `offload_adamw`, the DPO/GRPO/PPO stage scripts, P0-12 (Triton FA2 gate), P0-14 (adamw8bit at 1.8B local elements; this run used 1.4M).
- Timings are single-GPU, single-layer, random weights, with an idle foreign context resident on the card; they are not an end-to-end MFU and do not include FSDP gathers, attention, the optimizer or comms.
- `agree` figures compare loop vs grouped_mm, not either backend vs the HF reference (that is the oracle tier, CPU).
- Bit-exactness in (a) is for 2 ranks on this tiny model; a different kernel mix (flash/mem-efficient attention backward, larger embeddings) could introduce run-to-run noise that this setup did not exhibit (two uninterrupted runs were bit-identical).


---

# Track M: is align an OLMoE repo on the REAL checkpoint? (allenai/OLMoE-1B-7B-0125, 2026-10-12)

Everything below was measured on the released weights, not on a tiny model. Raw files: `.tmp/olmoe/` (`parity/`, `parity_post/`, `parity_trained/`,
`smoke_prefix_run/` = first 20-step pair, `smoke/` = final pair, `smoke_trim/`, `probe_*/`, `det_test.json`, `gradvar_before_fix.json`); the code that produced them is
`scripts/olmoe_parity.py`, `scripts/olmoe_sft_smoke.py` (subcommands `prepare`, `train`, `probe`, `drive`, `analyze`) and `configs/olmoe_sft_smoke.toml`; the pins are
`tests/test_olmoe_scripts.py`. Nothing here is an estimate unless it says so. The run sequence matters: the first 20-step pair ("pre-fix") exposed a bug that was then fixed in
`align/model/moe.py` (section f); the whole pair was then repeated on the fixed code ("final"). Numbers are from the final pair unless marked pre-fix.

## Environment

| Item | Value |
|---|---|
| Host / date | `pl-gpt-machine`, 2026-10-12 18:20 to 22:10 IST. 186 GB RAM, no swap. Preflight before every heavy step (MemAvailable >= 45 GB, load < 96, disk queue < 64, D-state <= 5, GPUs < 2 GiB used) and the heavy-job flock for every run; another track's `lean_ceiling_test` held the flock for ~15 min mid-session and my next job was refused (exit 83) rather than overlapping, then waited. |
| GPUs | 4x A10G, 23,028 MiB, all four **idle (0 MiB)** (the foreign 12 GiB `nmo_trace_server` of the first GPU pass was gone). PCIe, every pair `PHB`; **NCCL chose its `SHM` host-memory transport, not P2P** (`Channel ... via SHM/direct/direct`), and `NCCL_P2P_LEVEL=SYS` changed nothing. NCCL 2.28.9, torch 2.11.0+cu128. |
| Checkpoint | `allenai/OLMoE-1B-7B-0125` @ `9b0c1aa8`, 6 safetensors shards, **stored fp32** (27.68 GB, `torch_dtype: float32`; the brief's "~14 GB" is the bf16 size), 3,219 tensors, per-expert hub keys. Downloaded with `huggingface_hub` in the legacy venv (3 min 32 s; 26 GiB on `/apps`): safetensors + config + tokenizer only. |
| Oracle | transformers 5.15.0 in `/apps/development/rlhf/nano-mistral-o1/.venv`, subprocess only (the legacy-side code is a source string; `tests/test_dependency_firewall.py` passes). |
| Data (smoke) | the repo's cached PRM800K SFT shards (`nano-mistral-o1/data/shards/sft/train`, 11,739 Mistral-tokenized rows) decoded with the vendored Mistral tokenizer, split at `prompt_len`, **re-tokenized with the OLMoE tokenizer**; 320 rows of 640-968 tokens (mean 757; prompt mean 111), EOS-closed, prompt masked. Real data, not synthetic. |

## (e) Parity on the real checkpoint (CPU, align vs transformers 5.15, separate processes)

3 fixed prompts tokenized by the HF tokenizer (5, 39 and 30 tokens = 74 positions: prose, Python, prose with numbers); ids go to align. align `loop` + `sdpa`, transformers `eager`.
Each engine saves logits and per-layer router logits; a third step compares them.

| dtype | max abs logit diff | mean abs | top-1 | router top-8 **set** agreement, min over 16 layers | verdict |
|---|---|---|---|---|---|
| **fp32** | **1.4e-05** (sequential `index_add_` combine, pre-fix) / **3.0e-05** (fixed-order combine, post-fix); \|logit\| up to 25.3 | 1.1e-06 to 1.4e-06 | 74/74 | **1.00000 on every layer**; max router-logit diff 9e-06 | **PASS** (gate: < 1e-3, 100%, >= 99.9%) |
| bf16 | 1.31 | 0.048 to 0.064 | 74/74 | 0.892 to 0.973 per layer | **recorded, not gated** |

**Gate chosen: fp32, max abs < 1e-3 AND top-1 100% AND per-layer router set agreement >= 99.9%** (`olmoe_parity.evaluate`, pinned both ways in tests). Why not the brief's alternative
(bf16, top-1 100% with max abs < 0.1): the weights are stored fp32, so fp32 compute is the same function and any residual is accumulation order (measured 1.4e-05 to 3.0e-05, 30x to 70x under the
gate); and the bf16 alternative is **unsatisfiable by two correct implementations** (measured max abs 1.31: bf16 logits of magnitude 16-32 have an ulp of 0.125; align runs its router in fp32 while
transformers' bf16 router logits differ from it by up to 0.34, flipping 3-11% of top-8 sets). A bf16 tolerance that passes would be fitted to the answer. bf16 is run and recorded, never gated.

**The gate has teeth on these weights** (the same align code with one classic porting bug injected, compared with the same transformers tensors; identical before and after the combine fix):

| Injected defect | max abs logit diff | top-1 | min router set agreement | passes |
|---|---|---|---|---|
| none | 1.4e-05 to 3.0e-05 | 100% | 100% | yes |
| QK-norm per head instead of over the full projection | 11.6 | 81% | 5.4% | no |
| Mistral RoPE base 1e6 instead of 1e4 | 7.2 | 95% | 25.7% | no |
| expert gate/up stacked in the wrong order | 32.7 | 19% | 0.0% | no |
| Mixtral routing (top-k, then softmax over the kept experts) | 16.8 | 47% | 2.7% | no |

Pinned against the **released files** without loading weights (`test_real_checkpoint_config_is_the_documented_architecture_and_its_index_maps_totally_onto_the_model`): the hub `config.json`
(rope spelled the 4.x way, fp32 dtype) reads as the 6.919B architecture; all 3,219 index keys are consumed by `hf_to_align` and every parameter of the model is produced with the right shape (checked
on meta tensors built from the safetensors headers); index `total_size` = 4 x 6,919,161,856. The slow+oracle test (`-m "slow and oracle"`) re-runs the whole parity table and the four mutants (passes, 107 s).

**Round trip of a TRAINED checkpoint.** The checkpoint align's Trainer wrote at micro 80 of the SIGTERM-resumed run (bf16 weights, `config.json`, tokenizer files, 13.84 GB, gathered from FSDP2 shards by
rank 0) loads in transformers 5.15 and gives the same logits as align loading the same files: fp32 compute, max abs **1.43e-05**, top-1 74/74, router sets 100% (`parity_trained/`). The
"an align checkpoint loads in transformers" claim therefore holds at 7B-MoE scale through the real save path.

Cost: each engine loads in ~6 s (page cache hot) and runs the 3 prompts in ~2.9 s on 8 threads; peak RSS 51 GB (align) / 50 GB (transformers) at fp32, mostly mapped checkpoint pages.

## (f) A bug only the real weights could show: the MoE combine was nondeterministic (FIXED in `align/model/moe.py`)

**Symptom.** In the first 4-GPU pair the pre-clip global grad norm of optimizer step 1 was **5.44, 8.09 and 11.65 in three runs with identical weights, rows and seed** (the losses at micro 1-4 agreed
to 1e-3). Repeating the exact step-1 accumulation window four times inside ONE process on frozen weights (`probe --only gradvar`) gave **5.41, 10.21, 5.42, 13.28** with rank-mean losses
0.4216 / 0.4211 / 0.4214 / 0.4221: forward and backward were not reproducible.

**Isolation** (`.tmp/olmoe/det_test.py`: the real layers 0-3 + embeddings on one A10G, bf16, one batch, 5 repeats, hash of every layer's top-8 expert ids):

| variant | distinct losses / 5 | distinct router top-8 hashes / 5 | distinct layer-0 grad hashes / 5 |
|---|---|---|---|
| as shipped (`sdpa`, `index_add_` combine) | 5 | **5** | 5 |
| as shipped, eager attention | 5 | **5** | 5 |
| fixed-order combine, `sdpa` | **1** | **1** | 5 (attention-backward noise, norm 42.2606 to 42.2635) |
| fixed-order combine, eager attention | **1** | **1** | **1** |

**Cause.** `moe_experts_forward` summed a token's k=8 expert outputs with `out.index_add_(0, token_idx, y)` and gathered tokens with `x.index_select(0, token_idx)`, whose autograd backward is another
`index_add_` of k rows. On CUDA both are atomic scatters; in bf16 every partial sum is rounded, so the order the SMs happen to run in changes the hidden state by an ulp, and the router, sitting on
near-ties, then **selects different experts for the same tokens**. CPU `index_add_` is sequential and (measured) accumulates at higher precision; the oracle tier compares fp32 on CPU; so neither could see it.

**Fix.** `combine_experts` (un-sort to (token, slot) order with the plan's inverse permutation, then one `sum(dim=1)`: fp32 accumulation, one rounding, fixed order) and `_DispatchGather` (a custom
autograd function whose backward is the same fixed-order reduction). Equal to the old code in fp32 up to summation order; more accurate than 8 sequential bf16 adds in bf16; no measurable speed
change (compute-only 16-layer fwd+bwd, 1,024 tokens: 0.621 s before and after). After the fix:
- one MoE layer at OLMoE shape (E=64, k=8, T=4096, bf16) is **bit-identical over 8 repeats** on the A10G (`test_gpu_moe_layer_is_bit_reproducible_across_repeats_in_bf16`; the old code fails it);
- the FSDP step-1 window gives **14.42, 14.38, 14.35, 14.34** with bit-identical losses (0.42198020964860916 x4); the 0.6% residual is the SDPA mem-efficient backward (gradient noise, no routing effect);
- the old values (5.4 to 13.3) were not noise around 14.4: bf16 atomics under-reported the step-1 gradient norm by up to 2.7x;
- two full 20-step runs now agree on the step-1 norm to 0.2% (14.33, 14.30), and on per-step mean losses to <= 0.0016 (the pre-fix pair: <= 0.0096).

Regression tests (`tests/test_olmoe_scripts.py`): fixed-order exactness against an fp64 sum rounded once, an emulated-atomic bf16 combine that makes the same property fail on CPU, the slot-order
permutation test through both backends, structural pins that `moe_experts_forward` goes through the helpers and that the gather has the custom backward node, and the GPU repeat test. Six mutants of `moe.py`
(inline `index_add_`, sequential bf16 reduce, missing un-sort, wrong inverse, plain `index_select`, inline combine) are all killed. Oracle tier after the fix (`-m "oracle and not gpu"`): 17 passed, 6 skipped, the README baseline; full CPU suite
1600 passed, 13 skipped (other tracks' tests included).

**Still open: grad-norm excursions.** Even after the fix, three of 20 optimizer-step grad norms differ between the two final runs by 1.2x to 2.4x with per-micro losses equal to 1e-3 (step 11: 1.32 vs 1.77,
step 12: 1.42 vs 3.35, step 16: 1.70 vs 1.45); the other 17 agree to <= 4%. They are not caused by the resume (step 11 was logged before the stop; step 16 differs in the other direction). Replaying step 12's
window four times from the saved micro-44 weights gives 3.350 / 3.346 / 3.347 / 3.341: from fixed weights the norm is reproducible to 0.3%, so the uninterrupted run reached the same window with
weights that differ by bf16-level noise and got 1.42. The gradient norm is a discontinuous function of the weights (consistent with router top-k flips), not the output of a nondeterministic kernel.
Control: the following window (micro 49-52) from the same weights gives 1.3189 / 1.3187 / 1.3190 / 1.3188 (+-0.02%).

## (g) 4-GPU post-training smoke: FSDP2, bf16, adamw8bit, per-rank optimizer state, LBL off, telemetry on

`torchrun --nproc_per_node=4 scripts/olmoe_sft_smoke.py train --config configs/olmoe_sft_smoke.toml`: `OlmoeForCausalLM` (6.919B params, `loop` dispatch, `sdpa`), non-reentrant activation
checkpointing on all 16 blocks, `fully_shard` per block + root (bf16 params, fp32 reduce), `MoETrainModule` with `lb_weight = z_weight = 0` and per-layer telemetry on, fused chunked SFT cross-entropy,
micro-batch 1 x seq 1024 (every micro-batch padded to 1024) x accum 4 x 4 ranks = 16,384 padded tokens (12,080 real) per optimizer step, lr 1e-5 (10% warmup, cosine), clip 1.0, `adamw8bit`
(3.23 GiB of state per rank), 20 optimizer steps, checkpoint at step 10. Ranks draw distinct rows (rank stride over one seeded permutation) and load the model one at a time.

| Quantity | Measured (final pair; the pre-fix pair agrees to the digit unless noted) |
|---|---|
| Micro-step time, median of 77 intervals, 4 ranks in lockstep | **14.79 s** (min 14.60; max 16.89 = a micro-step containing an optimizer step); optimizer step = **59.2 s** |
| Throughput | **277 padded tokens/s** (4 GPUs), **204 real tokens/s**, 175 loss-tokens/s; 61 optimizer steps/h of 16 k padded tokens (the 20-step run is 20 minutes of training) |
| Achieved model FLOPs | 277 x 6 x 1.282B = **2.1 TFLOP/s total, 0.53 TFLOP/s per GPU** (~0.8% of an A10G's 70 TFLOP/s bf16; the roadmap assumed 20-25 TFLOP/s per card) |
| GPU memory per rank | allocator peak **14.83 GiB**, reserved peak **17.79 GiB**, `nvidia-smi` peak **18.5 GiB** (18,932 MiB on rank 0) of 22.5 GiB (23,028 MiB): **4.0 GiB headroom** on rank 0 (rank peaks 18,932 / 18,900 / 18,667 / 18,667 MiB; the allocator peaks are identical on all ranks). [Review 2026-10-12: this said 3.6 GiB, a MiB/GiB unit slip: 23,028 - 18,932 = 4,096 MiB.] After wrap + optimizer init 6.45 GiB (3.22 params + 3.23 8-bit state). The load peaks at 13.1 GiB (the full bf16 model on the card before sharding). |
| Mean loss per optimizer step (16 micro-batches each) | 0.464, 0.377, 0.361, 0.271, 0.247, 0.326, 0.169, 0.304, 0.361, 0.210, 0.247, 0.220, 0.219, 0.366, 0.254, 0.299, 0.410, 0.188, 0.288, 0.178 (first-5 mean 0.344, last-5 mean 0.273). Low from the start (the base model has seen this kind of math) and noisy (16 rows/step): it goes down; it is not a convergence claim. |
| Pre-clip grad norm | 14.33 (step 1), 3.66, 2.45, 1.98, 1.90, 2.26, 1.30, 1.71, 1.80, 1.28, 1.32, 1.42, 1.26, 1.53, 1.32, 1.70, 1.74, 1.20, 1.43, 0.99 (step 20). Above the 1.0 clip on 19 of 20 steps. Identical on all four ranks. `GradNormTripwire` ceiling is 500. |
| Skips / halts | 0 / none. Trunk-movement probe passed; every parameter, including all expert tensors, moved. |

**Router telemetry with load balancing OFF** (exact window values at the optimizer-step boundary; 64 experts, top-8; MaxVio = (max load - mean)/mean; utilization = sum_i min(f_i, 1/E)):

| layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MaxVio, step 1 | 4.75 | 5.33 | 2.95 | 2.18 | 6.23 | 6.32 | 5.57 | 5.54 | 6.15 | 6.21 | 5.48 | 6.28 | 5.00 | 4.24 | 3.82 | 6.18 |
| MaxVio, step 20 | 4.95 | 5.28 | 3.01 | 2.43 | 6.10 | 5.96 | 5.52 | 5.32 | 5.81 | 6.00 | 5.20 | 5.91 | 4.84 | 4.27 | 4.56 | 4.95 |
| utilization, step 1 | .757 | .769 | .724 | .707 | .661 | .617 | .563 | .570 | .631 | .601 | .644 | .587 | .653 | .651 | .639 | .635 |
| utilization, step 20 | .756 | .754 | .719 | .716 | .667 | .598 | .580 | .575 | .632 | .607 | .647 | .592 | .657 | .637 | .659 | .607 |

Layer-mean MaxVio **5.14 at step 1, 5.01 at step 20** (range 4.72 to 5.14 over the run; per-layer drift 0.3 to 1.3), worst layer 6.3 / 6.1, utilization 0.65 (worst layer 0.56 / 0.58), worst-layer dead-expert
fraction (< 0.1x mean load) **15.6% -> 12.5%** (up to 10 of 64 experts), router entropy 3.80 -> 3.79 nats (log 64 = 4.16), the load-balance loss VALUE (not applied) 1.50 -> 1.49 (1.0 = balanced), the z-loss
value (not applied) 8.6 -> 8.5. The pretrained OLMoE router is **already strongly skewed on this narrow math distribution (the busiest expert gets ~6x its fair share in most layers)**, and 20 steps of
LBL-off SFT at lr 1e-5 **do not move it** (no collapse, no drift beyond noise). That is evidence that LBL-off is safe for a short SFT, not for a long one, and says nothing about LBL-on (no A/B was run). Each telemetry point is one optimizer-step window all-reduced over the 4 ranks (16 distinct rows, ~12k real tokens), so a single point carries sampling noise from which problems happened to be in the window: the step-1 vs step-20 comparison is two such samples, and the 4.72-5.14 range over all 20 windows is the honest noise band.

**Checkpoint save and host memory (the 45 GB floor question).** A checkpoint is 27.7 GB: `model.safetensors` 13,838,724,744 B (bf16, per-expert hub layout, tokenizer files copied in) + 4 x 3,466,699,433 B of per-rank 8-bit optimizer state.

| | Before `malloc_trim` (pre-fix pair) | After (final pair and `smoke_trim/`) |
|---|---|---|
| Save wall time | 230-233 s | 228-241 s idle; 347 s once while a 3-minute CPU test suite ran on the same host (optimizer files done after ~2 min, `model.safetensors` ~2 min later) |
| Rank-0 RSS during a save | 2.1 -> **30.3 GB** (+28.1 GB = the gathered 13.8 GB bf16 state + the 13.8 GB per-expert clones `align_to_hf` makes) | 2.07 -> **30.2 GB** (unchanged: the transient is real) |
| Ranks 1-3 steady RSS | **14.3 GB each**, 13.4 GiB of it one anonymous `[heap]` (the freed bf16 state dict from the rank-serial load, stranded in glibc's brk heap; rank 0 kept ~11 GB after each save) | **2.0 GB each**; rank 0 2.06 GB after every save |
| Sum of the four ranks' RSS | peak 73.5 GiB (pre-fix ref), 89 GiB (resume) | peak **46.1 GiB** (a save), **90 GiB (resume)**: the resume reads four full CPU state dicts and is not helped |
| System MemAvailable | 141 idle -> 97 training -> **67.8 GB min** (during a save) | 134.6 idle/training -> **96.6 GB min** at saves; **70.1-75.9 GB min** around the resume and the final save of the SIGTERM run (another tenant held ~35 GB then) |
| Model load, per rank | 26.1-26.6 GB peak RSS (13.8 GB bf16 + the 13.8 GB stacked copy `hf_to_align` builds), 11-65 s; four ranks at once would be ~106 GB | same peak, then RSS falls to **0.74-0.98 GB** (`trimmed: true`) |
| `keep_checkpoints = 2` | not honoured across a resume: the SIGTERM run left 3 checkpoints (40, 44, 80 = 83 GB), because pruning only knows checkpoints written by the current process | same in the recorded run. **FIXED in review** (`Trainer.resume_from` now adopts the lower-numbered checkpoints on disk; higher-numbered zombies are still ignored; `tests/test_control_plane.py::test_resume_adopts_older_checkpoints_...`). Not re-run on GPUs. |

`trim_host_heap` (`malloc_trim(0)`) is applied after the load and after every save, outside the save's RSS window; pinned by an AST test and 3 mutants. **Added in review, NOT re-measured on GPUs:** a trim after `resume_from_latest()` too. In the recorded SIGTERM run ranks 1-3 sat at 13.7 GB RSS from the resume until the first save (`smoke/sup/ckpt_rss_rank1.jsonl`: `rss_start_gb` 13.73 at micro 80), the same stranded-heap shape as after the initial load. It does not change the 90 GiB summed peak DURING the resume read.

**SIGTERM, supervisor, resume** (`scripts/supervise.py` -> `torchrun`; SIGTERM to **rank 1's pid only**, 0.2 s after the step-10 checkpoint was complete on disk):

| Event | Observed (final pair; the pre-fix pair is identical in structure) |
|---|---|
| Stop | all four ranks logged `coordinated stop (signal) at optimizer step 11 (micro 44)`: the next boundary, one accumulation window (4 micro-steps, 59 s) later; stop checkpoint `micro_00000044` written (236 s) |
| Supervisor | attempt 1: child exit 1 (torchrun collapses the 143s), `progressed: true`, newest complete checkpoint 44; backoff 10 s; preflight ok; **attempt 2 launched, resumed at micro 44 / optimizer step 11**, ran micro 45-80, exit 0, "success" after 2 attempts. 2,072 s total. |
| Cost of the interruption | last micro-44 log -> first micro-45 log = **331 s** (236 s stop save + 10 s backoff + ~85 s relaunch, serial load and resume) against 14.8 s for a normal micro-step |
| Loss continuity vs the uninterrupted reference (same code, rows, seed) | micro 1-44 (same computation, two runs): max abs dloss **2.9e-03**, mean 7.0e-04 (the run-to-run floor). Micro 45-80 (resumed): max **3.3e-03**, mean 1.1e-03. Rule: resumed max <= 2 x floor + 5e-3 = 1.09e-02 -> **PASS** with 3.3x room. Per-step mean losses of the two runs differ by <= 0.0016. (Pre-fix pair: floor 1.2e-02, resumed 4.7e-03; the combine fix tightened the floor 4x.) **Review correction:** the original text here claimed 'a wrong optimizer shard or data position moves this loss by 0.1 to 1'. That was never measured (no mutant resume was run on the GPUs) and is not credible for a lost optimizer state at lr 1e-5 over 36 micro-steps, so the 1e-2 tolerance is a continuity check of unknown sensitivity. What actually establishes that this was a real resume is now checked exactly by `olmoe_sft_smoke.py analyze` (see 'Review of Track M' below). |
| Gradient norm at the resume | see "grad-norm excursions" in (f): the first resumed step (12) logged 3.35 against 1.42 in the reference, and replaying that window from the checkpoint weights reproduces 3.35 |

## (h) FSDP cost vs tokens, compute, and expert parallelism

The micro-step is 14.8 s for ~0.6 s of compute. Three independent probes on the idle box (`olmoe_sft_smoke.py probe --only collectives|compute|scan`; `scan` is the real FSDP2 model, fwd+bwd, no optimizer):

| Probe | Result |
|---|---|
| **FSDP micro-step time vs tokens** | L=256: **14.60 s**; 1,024: **14.72-14.77 s**; 2,048: **14.92 s**; 4,096: **15.02 s**; 8,192: **15.28 s**. Peak allocator memory 10.96 / 11.04 / 11.16 / 11.40 / **12.12 GiB** (+3.23 GiB for the 8-bit state in training: 15.4 GiB at 8,192 by that arithmetic. [Review: the training run's 14.83 GiB at 1,024 tokens is 0.56 GiB ABOVE this probe's 11.04 + 3.23 = 14.27 (gradient-accumulation and optimizer transients the scan does not have), so the honest 8,192 figure is ~15.9 GiB allocated, ~19.6 GiB `nvidia-smi`. It still fits, with ~2.9 GiB to spare; it is no longer 'comfortable'.]). Time is **flat in tokens**: +0.08 s per extra 1,024 tokens; ~14.5 s is fixed. |
| **NCCL on this box** (one block = 419.6 M params; bf16 all-gather of the block, fp32 reduce-scatter; un-overlapped) | all-gather **0.209 s** (4.0 GB/s algbw), reduce-scatter **0.430 s** (3.9 GB/s); the same at 1/16, 1/4 and full block size and with `NCCL_P2P_LEVEL=SYS`: it is the shared-memory transport's bandwidth, not latency. Per micro-step: 16 x (2 all-gathers [forward + activation-checkpoint recompute] + 1 reduce-scatter) = **13.55 s = 92% of the measured 14.8 s**. |
| **Compute only** (truncated random-init model on one GPU, no FSDP, activation checkpointing on): t = fixed + n_layers x per-layer, 2- and 4-layer fits | `loop` dispatch: 1,024 tokens 35.9 ms/layer + 47 ms -> **0.62 s for 16 layers**; 4,096 tokens 86.3 ms/layer + 128 ms -> **1.51 s**. `grouped_mm` (the torch fallback on sm_86): **0.44 s** (-30%) and **1.42 s** (-5.6%). |
| **Expert-parallel all-to-all** (bf16 activations, T x k x d per rank; 6 per layer per micro-step = forward + recompute + backward) | 1,024 tokens: 9.1 ms each (3.7 GB/s), **0.87 s per micro-step**; 4,096 tokens: 36.4 ms each, **3.50 s per micro-step** |

A **128k-token optimizer step** (32,768 tokens per rank). FSDP columns use the measured micro-step times above; compute and EP columns are arithmetic on measured per-op times, not runs (the 8,192-token compute 2.7 s and all-to-all 7.0 s are linear extrapolations of the 1,024/4,096 points; EP adds ~0.5 s per
step for the dense-parameter all-reduce, assumed non-overlapped):

| tokens/micro-batch | micro-steps/step | FSDP step (measured t_micro x n) | compute alone | FSDP / compute | EP-4 step (est.) | EP / compute | FSDP tok/s | EP tok/s (est.) | compute-bound ceiling |
|---|---|---|---|---|---|---|---|---|---|
| 1,024 | 32 | 32 x 14.72 = **471 s** | 32 x 0.62 = 19.8 s | **24x** | 32 x (0.62+0.87)+0.5 = 48 s | 2.4x | 278 | ~2,700 | 6,600 |
| 4,096 | 8 | 8 x 15.02 = **120 s** | 8 x 1.51 = 12.1 s | **10x** | 8 x (1.51+3.50)+0.5 = 41 s | 3.4x | 1,090 | ~3,200 | 10,800 |
| 8,192 | 4 | 4 x 15.28 = **61 s** | 4 x ~2.7 (extrapolated) = ~11 s | **5.6x** | 4 x (2.7+7.0)+0.5 = 39 s | 3.6x | 2,140 | ~3,400 | ~12,000 |

## What each number decides

| Number | Decision it feeds |
|---|---|
| fp32 logits 1.4e-05 to 3.0e-05, top-1 74/74, router sets 100% on all 16 layers; the trained checkpoint round-trips through transformers at 1.4e-05; four injected bugs fail by 5-6 orders | **P5 "HF logit parity" is met on the released weights; align is an OLMoE repo.** Gate = fp32 < 1e-3 / 100% / 99.9%. bf16 (1.31) is a recorded noise floor and must not be used as a gate. |
| bf16 router set agreement 89-97% between two correct implementations | Under bf16, **router replay (R3) and any flip-rate threshold must be measured against align's own sampler, never against transformers**: the HF-vs-align flip rate (3-11% per layer) is the baseline disagreement of bf16 routers, far above the roadmap's 0.1% skip-R3 heuristic. |
| Combine nondeterminism: 5/5 distinct top-8 sets for identical inputs before the fix, 1/1 after | Fixed. Any MoE flip-rate or mismatch-KL probe taken on the pre-fix code measured atomics, not engine mismatch: re-baseline. Residual: SDPA backward gives ~0.6% gradient noise (no routing effect), and the gradient norm is discontinuous in the weights (excursions in (f)). |
| Pre-clip grad norm 14.3 at step 1, > 1.0 on 19 of 20 steps | With Adam the clip does not shrink the update, it re-weights early steps against late ones (step 1 is down-weighted 14x). Log a clipped fraction next to any lr claim and compare against an unclipped baseline before attributing an SFT effect to the lr. |
| FSDP micro-step **14.8 s, flat in tokens**; 92% is NCCL over host shared memory at ~4 GB/s | **Roadmap gate 3 (FSDP vs EP): exceeded.** FSDP-only is 24x / 10x / 5.6x compute at 128k tokens per step with 1,024 / 4,096 / 8,192-token micro-batches; the criterion was 3x. Build EP (P1-13) if ~1,000-2,000 tokens/s is not enough. |
| EP all-to-all 3.5 s vs FSDP 13.55 s per micro-step at 4,096 tokens (16x less at 1,024; 2x less at 8,192) | EP is worth 3x at 4,096-token micro-batches and 1.6x at 8,192, and is **still comm-bound (3.4-3.6x compute) on this PCIe-SHM box**. The **free lever is the micro-batch**: 1,024 -> 8,192 tokens takes plain FSDP from 278 to 2,140 tokens/s. The 4-GPU ceiling for OLMoE post-training here is ~2-3.4 k tokens/s = 0.2-0.3 B tokens/day, against the roadmap's 1.1 B/day (which assumed 25 TFLOP/s per card): re-state the OLMoE budget before pricing an SFT epoch or a proxy. |
| 277 tokens/s = 0.53 TFLOP/s per GPU at the shipped micro-batch; 4.2% of the 6,600 tokens/s compute-bound ceiling | Replaces the 25 TFLOP/s MFU assumption (P1-15). |
| `grouped_mm` -30% (1,024 tokens) / -5.6% (4,096) of compute, but compute is 4% of the FSDP micro-step | **Roadmap gate 2 (grouped GEMM) stays closed for FSDP-only training** (worth <= 0.2 s of 14.8 s). It becomes a 5-30% lever only after the comm wall is removed (EP or larger micro-batches). Keep `loop` as the default and the oracle. |
| Peak 14.83 GiB allocated / 18.5 GiB `nvidia-smi` of 22.5; +0.08 s per extra 1,024 tokens; 8,192-token micro-batches fit (~15.9 GiB allocated; see the correction under (h)) | Headroom is 4.0 GiB at 1,024 tokens and the model fits at 8,192. OLMoE-1B-7B full-FT with adamw8bit is **bandwidth-limited, not memory-limited**. |
| `adamw8bit` + FSDP2 on 3-D expert tensors (sharded on dim 0, 16 experts per rank): 20 steps, no skips, all parameters moved | Closes the "adamw8bit / FSDP2 with MoE `RouterStats` on GPU" part of P0-14 and of the `docs/ARCHITECTURE.md` limits: router stats ride out of non-reentrant-checkpointed blocks without double counting (telemetry stable across 80 micro-steps). |
| MaxVio ~5 (worst 6.3), utilization 0.65, 12-16% dead experts at step 1, unchanged at step 20 with LBL off | **Roadmap gate 7 (LBL off unless the load tripwire fires): not fired.** No MaxVio threshold is pre-registered yet; these are the first numbers to base one on. Because the base router is already this skewed on math data, an LBL-on arm would pull away from the pretrained solution: run that A/B before enabling it. |
| Save 228-241 s idle; rank-0 peak RSS 30.2 GB (+28 GB); MemAvailable min 67.8 GB before the heap fix, 96.6 GB after; four ranks 90 GiB during a resume | **Roadmap gate 4 (checkpoint RSS):** the criterion was a per-rank save peak within ~10 GB of the 45 GB floor. It is not met (30.2 GB; 15 GB under), so reshardable checkpoints are **not forced into P0** by this measurement. The job's own footprint was the bigger risk: 40 GiB of dead heap on ranks 1-3, now fixed (`malloc_trim`). Remaining: the 28 GB save transient (needs a per-tensor streaming writer) and the 90 GiB resume (four full CPU state dicts: needs a shard-local read). With both, one OLMoE job coexists with a second ~60 GB job; today it coexists with ~35 GB. |
| SIGTERM to one rank -> all four stop at the same boundary -> supervisor relaunch -> loss within 3.3e-3 of the uninterrupted run (floor 2.9e-3) | Moves P0-1 / P0-8 / P0-9 from "2-rank NCCL" to **"4-rank NCCL, 6.9B model, real save/resume" verified**. Budget for the SIGTERM grace period: boundary latency 59 s + 236 s for the stop checkpoint; the whole interruption cost 5.5 minutes. |

## Findings that were not asked for (Track M)

1. **`scripts/sft.py` / `scripts/pretrain.py` (dense) give every rank the same batch stream**: `build_training_sampler` returns one unsharded `LengthGroupedSampler` and the Trainer does no rank slicing (only
   `dpo_stage` wraps in `DistributedBatchSampler`). By code reading, a 4-GPU dense SFT trains 4 identical copies of each batch: the reduced gradient equals one rank's, the configured 24-row
   "global batch" is 6 rows, and the horizon arithmetic is 4x too long. Not verified by a run (no dense 7B weights on this box). The OLMoE smoke shards by rank (`RankSampler`, disjointness pinned).
2. **`keep_checkpoints` leaks one checkpoint per resume** (13.8 + 13.9 GB): pruning is driven by a per-process list. **Fixed in review** (see the table above).
3. **The `history.jsonl` collision (P0-6) is still live** in the Trainer; the smoke gives ranks 1-3 their own `rank{r}/history.jsonl` and rank 0 the one the supervisor watches.
4. **The checkpoint is fp32 on disk (27.7 GB)**, twice the brief's "~14 GB"; `load_hf_safetensors(dtype=bf16)` casts shard by shard so a bf16 load never holds both.
5. **`/apps` is 94-95% full.** One SIGTERM/resume pair of 20-step runs writes ~130 GB of checkpoints; all checkpoints from these runs were deleted after analysis (histories, telemetry and logs kept).
6. **Host-memory fix beyond the brief**: `malloc_trim` after load and save recovered ~40 GiB of dead glibc heap on a no-swap box (above). It applies to any rank-serial 7B load in this repo, dense included.
7. The shared heavy-job lock did its job against another track's long job (refused with exit 83, then waited).
8. `torch` could not create `.cache/torch/kernels` (a warning on every MoE entropy call); the empty directory now exists. Untracked: `.cache/`.

## Not measured / still open (Track M)

- LBL-on vs LBL-off A/B; a router flip rate between a sampler and the trainer (needs the MoE decode path, P3-1); `grouped_mm` end to end under FSDP; EP itself (only its all-to-all volume was timed);
  a 128k-token step (largest micro-batch run: 8,192 tokens, forward+backward only); runs longer than 80 micro-steps; `kill -9` mid-save; a second large job alongside (the 45 GB floor under real contention);
  GRPO / DPO on OLMoE.
- Throughput numbers are for `loop` dispatch with `sdpa`; `grouped_mm` and other attention variants were timed compute-only.
- The grad-norm excursions of (f) are characterised (reproducible from fixed weights, not from a kernel) but not traced to a specific router flip.

---

# Review of Track M (independent reviewer, 2026-10-12)

What was re-checked, how, and what changed. Raw: `.tmp/review/` (`parity1/`, `parity1.log`).

| Claim | Check | Result |
|---|---|---|
| fp32 parity on the real checkpoint | Re-ran `olmoe_parity.run` (CPU, separate processes, current working tree incl. the fixed-order combine) on **one prompt that is not in the original three**: "The sum of the first n integers is n(n+1)/2; for n = 100 that gives 5050, and 17 * 23 = 391." (32 tokens, \|logit\| up to 23.1) | **max abs 1.57e-05**, mean 1.38e-06, top-1 32/32, router top-8 set agreement **1.00000 on all 16 layers** (min overlap 8/8), max router-logit diff 9.4e-06. Gate PASS. Consistent with the recorded 1.4e-05 / 3.0e-05. Both workers took the heavy-job lock; peak RSS ~51 GB; MemAvailable was observed at >= 72 GB while it ran. |
| Recorded parity tables | Recomputed from `parity/`, `parity_post/` `report_*.json` and `mutants_fp32.json` | All numbers in section (e) match the raw files (fp32 1.383e-05 / 2.956e-05, bf16 1.306, 0.892-0.973, the four mutants). |
| LBL off, telemetry on | `launch_rank*.json` of every attempt of both runs; `history.jsonl` keys | `lb_weight = z_weight = 0.0` on every rank of every launch; `moe/l{0..15}/max_vio`, `utilization`, `moe/lb_loss`, `moe/z_loss` logged every row; `optimizer` banner = adamw8bit on-device; `moe_backend = loop`. |
| The resume was a real resume | `sup/launch_rank*_*.json`, raw `history.jsonl` per rank, `supervisor.jsonl` | Every rank's 2nd launch: `resumed_from = .../checkpoints/micro_00000044`, `resumed_micro_step = 44`; first launches `None / 0`. Each rank's raw history is micro 1..80 exactly once (80 rows, no duplicates, no restart). Supervisor: attempt 1 exit 1 `progressed` ckpt 44, attempt 2 exit 0 ckpt 80. |
| Same data position | batch row ids of all 4 ranks x 80 micro-steps, ref vs sup | **320/320 identical**, and `lr` identical at micro 40/44/48/80. |
| Per-rank optimizer files used | `per_rank_optimizer_state = true` in the config (pinned by test); trainer writes `optimizer_rank{r}.pt` and `checkpoint_problem` requires one per rank (`align/train/trainer.py`); each rank's `ckpt_rss_rank{r}.jsonl` shows rank 0 alone at a 30.2 GB peak and ranks 1-3 at 5.25 GB (a 3.5 GB local write, no gather); the 4 x 3,466,699,433 B sizes match 6.919 B x 2.004 B/param / 4 | Consistent. The checkpoint directories themselves were deleted to free 130 GB, so the file names could not be re-listed; this is inference from the trainer code, the config and the RSS records. |
| Throughput and memory arithmetic | 4 x 1024 / 14.79 s = 277 padded tok/s; x 757/1024 = 204 real; x 646/1024 = 175 loss tokens; 277 x 6 x 1.282 B = 2.1 TFLOP/s = 0.53 per GPU; 3600 / 59.2 = 61 steps/h; step-time median over 77 intervals re-derived from `history.jsonl` (14.795 s ref / 14.797 s sup); alloc peak 14.828 / reserved 17.79 GiB identical on ranks 0-3; nvidia-smi peaks 18,932 / 18,900 / 18,667 / 18,667 MiB; FSDP/EP/128k table arithmetic redone | All consistent, **except** the headroom (3.6 GiB, should be 4.0) and the 8,192-token memory extrapolation (15.4 should be ~15.9): both corrected above. |
| Telemetry and loss numbers | Recomputed from rank-0 and per-rank histories: MaxVio 5.1396 -> 5.0053, worst 6.32 -> 6.10, dead fraction 0.156 -> 0.125, entropy 3.8035 -> 3.7915, lb 1.503 -> 1.4875, z 8.589 -> 8.512; per-step mean losses and first-5 / last-5 means (0.344 / 0.2726); grad-norm list; clip active on 19 of 20 | All match. |
| Resume loss tolerance | Re-ran `analyze` on the recorded pair | floor 2.93e-03, resumed 3.30e-03, tolerance 1.086e-02, PASS: matches the doc. |

## Defects found and fixed (each with a regression test; mutants that remove the fix fail)

1. **The resume verdict could PASS on a resume that never happened** (`scripts/olmoe_sft_smoke.py`). `analyze` de-duplicated history by micro-step (last wins) and compared loss curves; a relaunch that ignored the checkpoint and retrained from micro 0 yields a de-duplicated curve equal to the reference, so the verdict was PASS. Now `analyze` also requires `check_resume_provenance` (second launch loaded `micro_{stop}` and resumed at the stop step on every rank; raw history advances by exactly one per row) and `compare_batches` (every rank saw the same row at every micro-step as the uninterrupted run), and exits 1 otherwise. The recorded run satisfies all of it. Tests: `test_analyze_passes_a_genuine_resume_and_fails_a_restart_from_zero_...`, `test_provenance_flags_each_way_a_resume_can_be_fake`, `test_batch_compare_is_exact_...`, `test_analyze_fails_when_a_rank_saw_a_different_row_...`, `test_the_recorded_track_m_run_satisfies_the_provenance_gates`; 5 script mutants killed.
2. **The claimed sensitivity of the loss-curve gate was never measured** (see the correction in the resume table). The docstring and this document no longer say a wrong optimizer shard moves the loss by 0.1-1. **Still open:** a mutant resume on the GPUs (zeroed 8-bit moments; base weights instead of the checkpoint) to find out what the 1e-2 tolerance actually detects.
3. **No heap trim after the resume read** (13.7 GB stranded on ranks 1-3 until the first save). Trim added; AST test `test_heap_trim_is_also_wired_after_the_resume_read`; mutant killed. Not re-measured on GPUs.
4. **`keep_checkpoints` leaked one checkpoint per resume** (`align/train/trainer.py`, a file the track did not own): `resume_from` adopts the lower-numbered checkpoint directories; higher-numbered zombies are still ignored. Adopted steps go in FRONT of the resumed one because `scripts/ppo.py::_latest_weights` reads `checkpoints_written[-1]` as the newest (the first version of the fix appended them after and broke `tests/test_ppo.py::test_resume_restores_the_latest_weights_pointer`, caught by the full suite). `tests/test_trainer.py::test_resume_registers_the_resumed_checkpoint_in_checkpoints_written` used to pin exactly `[8]` and now pins `[4, 8]`. Test `tests/test_control_plane.py::test_resume_adopts_older_checkpoints_so_keep_n_does_not_leak_one_per_resume`; 2 mutants killed (no adoption; adopting zombies too).
5. **Doc arithmetic**: headroom 3.6 -> 4.0 GiB (MiB/GiB slip) and the 8,192-token allocator extrapolation 15.4 -> ~15.9 GiB. Telemetry-noise caveat added.

## Observations, not changed

- Resume on step 12 showed grad norm 3.35 against 1.42 in the reference even though the batch rows are identical (320/320) and the lr is identical: the difference is in the weights. The track's explanation (replaying the window from the saved micro-44 weights reproduces 3.35 to 0.3%) is consistent with that; it rules out a data-position bug and a kernel nondeterminism, and leaves "weights differ by bf16-level noise and the gradient is discontinuous in them". It does not by itself exclude a lossy checkpoint round trip (bf16 weights are authoritative under stochastic rounding, so there should be none), which only the mutant/identity test in item 2 would settle.
- `README.md`, `docs/ARCHITECTURE.md` and `docs/ROADMAP.md` status lines and the 25 TFLOP/s / 1.1 B tokens/day constants are stale relative to section (g)/(h); left to the owner, as the track reported.
- **Shared batch stream in dense `scripts/sft.py` / `pretrain.py`: confirmed by reading, still unfixed.** `build_training_sampler` returns one unsharded `LengthGroupedSampler`; `Trainer`'s `ResumableBatchStream` does `list(self.sampler)` with no rank slice (only `dpo_stage` wraps in `DistributedBatchSampler`). Every rank therefore trains on the same batch indices, FSDP averages four identical gradients, and `len(sampler)` (the LR horizon) is 4x the per-rank work. Not run (no dense 7B weights); not fixed here because the horizon arithmetic in the entry gates (`sampler_derived_horizon`) and `global_batch_rows` semantics must change together, which is the owner's call. Severity: high for any multi-GPU dense run.