File size: 3,925 Bytes
11f07f9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | # d24 bf16, total batch 2^20: upstream nanochat vs single-file -v3
All runs on the same 8xH100 box unless the label says GH200. `bf16_mfu` is **not** comparable across the two pipelines (different FLOPs/token estimates: upstream 4.775e9, ours 4.860e9) — compare **ms/step** and **tok/sec**, which measure the same 2^20 tokens per step.
| run | parsed | ms/step (steps 20-99) | tok/sec | bf16_mfu* |
|---------------------------------------|------------|-------------------------|-----------|-------------|
| ours w8 64K micro (full run) | 5568 steps | 1212 | 864,856 | 53.13 |
| ours w8 32K micro +FP8 | 200 steps | 1040 | 1,008,030 | 61.92 |
| ours w8 32K micro | 100 steps | 1249 | 839,838 | 51.59 |
| upstream w8 bf16 | 1017 steps | 1291 | 812,435 | 49.03 |
| upstream w8 +fp8 | 200 steps | 1081 | 970,455 | 4.86 |
| fork varlen branch w8 bf16 | 200 steps | 1278 | 820,783 | 49.54 |
| fork varlen+fp8 w8 (2026-03-22, full) | 5568 steps | 1066 | 983,648 | 59.37 |
| ours w1 (1xH100, partial) | 23 steps | 9620 | 109,002 | 53.57 |
| ours w1 (1x GH200) | 100 steps | 9635 | 108,834 | 53.48 |
| derived | value |
|-----------|---------|
## Correctness: does world=8 reproduce the world=1 curve?
The loader assigns every rank's documents from one sequential stream and total batch is fixed, so the global batch per optimizer step is identical at every world size — these are directly comparable.
| run | step 0 | step 25 | step 50 | step 75 | step 100 |
|--------------------------------|----------|-----------|-----------|-----------|------------|
| -v3 world=8 (8xH100) | 3.17614 | 1.75957 | 1.57767 | 1.45099 | 1.37943 |
| -v3 world=1 (GH200, committed) | 3.17614 | 1.75956 | 1.57762 | 1.4503 | 1.3794 |
| run | min val bpb | peak alloc MiB | peak reserved MiB | train time |
|-------------------------------------|---------------------|------------------|---------------------|--------------|
| -v3 world=8 | 1.379430 | 40049 | 65750 | 1.85m |
| -v3 world=1 GH200 | 1.379400 | 49283 | 71046 | 14.29m |
| world=1 reference band (eval100a-e) | 1.379318 - 1.379563 | 49,088 - 49,334 | -- | -- |
## Head-to-head val bpb (both on shard_06542)
| step | -v3 (full run) | upstream baseline | delta |
|--------|------------------|---------------------|---------|
| 0 | 3.17584 | 3.16882 | +0.22% |
| 250 | 0.997472 | 0.984017 | +1.37% |
| 500 | 0.904081 | 0.898168 | +0.66% |
| 750 | 0.866224 | 0.861772 | +0.52% |
| 1000 | 0.846058 | 0.841709 | +0.52% |
The upstream baseline was stopped at step 1000, so it bounds the comparison; beyond that the -v3 curve stands alone.
Full-run val bpb curve: 0:3.175835 / 250:0.997472 / 500:0.904081 / 750:0.866224 / 1000:0.846058 / 1250:0.832731 / 1500:0.822766 / 1750:0.815902 / 2000:0.809199 / 2250:0.798533 / 2500:0.789216 / 2750:0.781275 / 3000:0.773139 / 3250:0.766625 / 3500:0.759539 / 3750:0.753329 / 4000:0.747155 / 4250:0.741616 / 4500:0.736278 / 4750:0.731419 / 5000:0.726908 / 5250:0.722909 / 5500:0.719696 / 5568:0.719042
_Produced by `make_baseline_report.py`; sources listed in its docstring._ |