d24 bf16, total batch 2^20: upstream nanochat vs single-file -v3
All runs on the same 8xH100 box unless the label says GH200. bf16_mfu is not comparable across the two pipelines (different FLOPs/token estimates: upstream 4.775e9, ours 4.860e9) — compare ms/step and tok/sec, which measure the same 2^20 tokens per step.
| run | parsed | ms/step (steps 20-99) | tok/sec | bf16_mfu* |
|---|---|---|---|---|
| ours w8 64K micro (full run) | 5568 steps | 1212 | 864,856 | 53.13 |
| ours w8 32K micro +FP8 | 200 steps | 1040 | 1,008,030 | 61.92 |
| ours w8 32K micro | 100 steps | 1249 | 839,838 | 51.59 |
| upstream w8 bf16 | 1017 steps | 1291 | 812,435 | 49.03 |
| upstream w8 +fp8 | 200 steps | 1081 | 970,455 | 4.86 |
| fork varlen branch w8 bf16 | 200 steps | 1278 | 820,783 | 49.54 |
| fork varlen+fp8 w8 (2026-03-22, full) | 5568 steps | 1066 | 983,648 | 59.37 |
| ours w1 (1xH100, partial) | 23 steps | 9620 | 109,002 | 53.57 |
| ours w1 (1x GH200) | 100 steps | 9635 | 108,834 | 53.48 |
| derived | value |
|---|
Correctness: does world=8 reproduce the world=1 curve?
The loader assigns every rank's documents from one sequential stream and total batch is fixed, so the global batch per optimizer step is identical at every world size — these are directly comparable.
| run | step 0 | step 25 | step 50 | step 75 | step 100 |
|---|---|---|---|---|---|
| -v3 world=8 (8xH100) | 3.17614 | 1.75957 | 1.57767 | 1.45099 | 1.37943 |
| -v3 world=1 (GH200, committed) | 3.17614 | 1.75956 | 1.57762 | 1.4503 | 1.3794 |
| run | min val bpb | peak alloc MiB | peak reserved MiB | train time |
|---|---|---|---|---|
| -v3 world=8 | 1.379430 | 40049 | 65750 | 1.85m |
| -v3 world=1 GH200 | 1.379400 | 49283 | 71046 | 14.29m |
| world=1 reference band (eval100a-e) | 1.379318 - 1.379563 | 49,088 - 49,334 | -- | -- |
Head-to-head val bpb (both on shard_06542)
| step | -v3 (full run) | upstream baseline | delta |
|---|---|---|---|
| 0 | 3.17584 | 3.16882 | +0.22% |
| 250 | 0.997472 | 0.984017 | +1.37% |
| 500 | 0.904081 | 0.898168 | +0.66% |
| 750 | 0.866224 | 0.861772 | +0.52% |
| 1000 | 0.846058 | 0.841709 | +0.52% |
The upstream baseline was stopped at step 1000, so it bounds the comparison; beyond that the -v3 curve stands alone.
Full-run val bpb curve: 0:3.175835 / 250:0.997472 / 500:0.904081 / 750:0.866224 / 1000:0.846058 / 1250:0.832731 / 1500:0.822766 / 1750:0.815902 / 2000:0.809199 / 2250:0.798533 / 2500:0.789216 / 2750:0.781275 / 3000:0.773139 / 3250:0.766625 / 3500:0.759539 / 3750:0.753329 / 4000:0.747155 / 4250:0.741616 / 4500:0.736278 / 4750:0.731419 / 5000:0.726908 / 5250:0.722909 / 5500:0.719696 / 5568:0.719042
Produced by make_baseline_report.py; sources listed in its docstring.