Update README: add V3 Val Loss/PPL results table, bf16+lr_scale finding

README.md

@@ -109,21 +109,29 @@ step checkpoints (sample of English narrative):
 
 ## Round 3 — cold-start rescue (interrupted)
 
-After the audit, configurations were updated to apply corrections such as **bf16 AMP** (instead of fp16), **`lr_scale=2.0` on BitLinear**, and an increased budget of **150 tok/param**.
-
-The current trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). None of them generated the final JSON. Extracted from the Hugging Face Hub (`Cukinator/cpu1-ablation-checkpoints-v3`), here are the completion percentages of each run before termination:
-
-| Run | Architecture | Params | Target Steps | Last Step | Progress |
-|-----|-------------|-------:|-------------:|----------:|---------:|
-| run_01 | Transformer + BPE (16K vocab) + FP16 | 54.7M | 62,500 | 12,173 | **19.5%** |
-| run_02 | Transformer + Byte + LocalByteDecoder | 38.8M | 62,500 | 10,963 | **17.5%** |
-| run_04 | MLGRU + Byte + Ternary | 38.9M | 62,500 | 6,074 | **9.7%** |
-| run_05 | + FPResidual | 39.0M | 62,500 | 4,390 | **7.0%** |
-| run_06 | + Bolmo patch embedding | 39.0M | 62,500 | 4,237 | **6.8%** |
-| run_08 | Folded Transformer + Byte + Ternary | 38.9M | 62,500 | 3,858 | **6.2%** |
-| run_13 | Small CPU-1 + BPE (4K vocab) | 12.5M | 12,500 | 9,290 | **74.3%** |
-| run_14 | Small CPU-1 byte-level (Qwen distillation) | 10.7M | 12,500 | 9,775 | **78.2%** |
-| run_15 | Small CPU-1 byte + hidden distillation | 10.7M | 12,500 | 2,350 | **18.8%** |
+After the audit, configurations were updated to apply corrections: **bf16 AMP** (instead of fp16), **`lr_scale=2.0` on BitLinear**, and an increased budget of **150 tok/param**.
+
+The trainings were interrupted prematurely due to cloud credit exhaustion (AMD MI300X, 192GB VRAM). Val Loss and Perplexity were measured from the interrupted checkpoints (evaluated on the last 3 shards of the validation set, 100 batches):
+
+### 50M runs (Round 3)
+
+| Run | Architecture | Params | Progress | Val Loss | Perplexity | Notes |
+|-----|-------------|-------:|---------:|---------:|-----------:|-------|
+| run_01 | Transformer + BPE (16K vocab) + FP16 | 54.7M | 19.5% (12,173 / 62,500 steps) | — | — | ❌ Diverged (NaN weights) |
+| run_02 | Transformer + Byte + LocalByteDecoder | 38.8M | 17.5% (10,963 / 62,500 steps) | — | — | ❌ Diverged (NaN weights) |
+| run_04 | MLGRU + Byte + Ternary | 38.9M | 9.7% (6,074 / 62,500 steps) | **1.3064** | **3.69** | ✅ Converging |
+| run_05 | + FPResidual | 39.0M | 7.0% (4,390 / 62,500 steps) | **1.3422** | **3.83** | ✅ Converging |
+| run_06 | + Bolmo patch embedding | 39.0M | 6.8% (4,237 / 62,500 steps) | **0.6261** | **1.87** | ✅ Converging |
+| run_07 | + DeleteGate (CPU-1 complete) | 39.0M | — | — | — | ⚠️ Checkpoint not found in V3 repo |
+| run_08 | Folded Transformer + Byte + Ternary | 38.9M | 6.2% (3,858 / 62,500 steps) | — | — | ❌ Diverged (NaN weights) |
+
+### 10M runs (Round 3)
+
+| Run | Architecture | Params | Progress | Val Loss | Perplexity | Notes |
+|-----|-------------|-------:|---------:|---------:|-----------:|-------|
+| run_13 | Small CPU-1 + BPE (4K vocab) | 12.5M | 74.3% (9,290 / 12,500 steps) | 25.01 | ~4.9×10⁸ | BPE cold-start collapse |
+| run_14 | Small CPU-1 byte-level (Qwen distillation) | 10.7M | 78.2% (9,775 / 12,500 steps) | **0.5305** | **1.70** | ✅ Strong convergence |
+| run_15 | Small CPU-1 byte + hidden distillation | 10.7M | 18.8% (2,350 / 12,500 steps) | **0.9696** | **2.64** | ✅ Converging |
 
 The extremely slow progress (<20% in 1 day for 50M models) was caused by a severe CPU context-switching bottleneck when orchestrating 16 concurrent processes on a single MI300X GPU.
 
@@ -141,18 +149,24 @@ at step 10 and at step 1326** (the entire training). The bottleneck is the
 cold-start dynamics of straight-through-estimator (STE) ternary
 quantisation, not the data budget.
 
-### V1 vs V3 Architecture: Isolating the Problem
+### V3 Key Finding: bf16 AMP + lr_scale=2.0 Breaks the Ternary Plateau
 
-When analyzing the ablation results, the classic V1 Transformer architecture in FP16 (`run_02`) works perfectly (Loss: 1.72, PPL: 5.56). However, the complete V3 architecture (`run_07`, CPU-1 with MLGRU, Bolmo, and DeleteGate) using ternary weights collapses to random noise (Loss: 5.56, PPL: 258.8).
+In V1/V2, **all** ternary runs (Round 1 and Round 2) were stuck at Loss ≈ 5.55 nats — the uniform entropy of the 256-byte vocabulary. The V3 recipe applied two fixes: **bf16 AMP** (replacing fp16) and **`lr_scale=2.0` for BitLinear**.
 
-To isolate the cause of the collapse, we compared the same V1 architecture but forced the use of ternary weights (`run_08`):
+Even with only 6–10% of target training completed, the V3 ternary runs show **real convergence** for the first time:
 
-| Model | Base Architecture | Weights | Loss | Perplexity | Result |
-|---|---|---|---|---|---|
-| `run_08` | V1 (Classic Transformer) | Ternary | 5.56 | 260.4 | ❌ Stuck |
-| `run_07` | V3 (MLGRU + Bolmo + DeleteGate) | Ternary | 5.56 | 258.8 | ❌ Stuck |
+| Run | Round | Architecture | Val Loss | Perplexity | Notes |
+|-----|-------|-------------|---------|-----------|-------|
+| run_04 | V1 | MLGRU + Byte + Ternary | 5.57 | 261.7 | ❌ Stuck at uniform floor |
+| run_04 | V3 | MLGRU + Byte + Ternary | **1.3064** | **3.69** | ✅ Real convergence |
+| run_05 | V1 | + FPResidual | 5.55 | 257.7 | ❌ Stuck |
+| run_05 | V3 | + FPResidual | **1.3422** | **3.83** | ✅ Real convergence |
+| run_06 | V1 | + Bolmo patch embedding | 5.56 | 258.8 | ❌ Stuck |
+| run_06 | V3 | + Bolmo patch embedding | **0.6261** | **1.87** | ✅ Real convergence |
+| run_14 | V1 | Small CPU-1 byte (10M) | 5.58 | 263.9 | ❌ Stuck |
+| run_14 | V3 | Small CPU-1 byte (10M) | **0.5305** | **1.70** | ✅ Strong convergence |
 
-**Conclusion:** Both models crashed against the same numerical wall (Loss = ~5.56, the uniform entropy of 256 bytes)! This completely absolves the V3 architecture (MLGRU/DeleteGate/Bolmo). The failure is exclusively due to the training mechanics of the 1.58-bit ternary weights (BitLinear) in a *cold-start* condition at this scale.
+**Conclusion:** The V3 recipe (bf16 AMP + 2× LR for BitLinear) successfully breaks the cold-start ternary collapse that plagued all V1/V2 runs. These results are from early-interrupted checkpoints at 6–78% of training — final trained models would be expected to achieve significantly lower loss.
 
 Published 1.58-bit models (BitNet b1.58 at 700M+ with 30–143 tok/param;
 Slender-Mamba with FP16 warm-start) reach functional performance, but the