Update to constant-LR refinement best checkpoint (epoch 163, val_loss 3.608)

README.md

@@ -28,35 +28,35 @@ replacement in the same vLLM serving path.
   FSDP training + vLLM rollout with cross-node hidden-state transfer over
   Mooncake (RDMA).
 - **Topology**: 1 datagen node (Kimi-K2.6, TP=8) -> Mooncake -> 1 trainer GPU.
+- **Schedule**: an initial 10k-step cosine phase (LR 2e-5), then a constant
+  low-LR (2e-6) refinement phase continued from the best checkpoint. This
+  published checkpoint is the best-val checkpoint of the refinement phase.
 - **Data**: online-generated hidden states from a ~100k-prompt instruction set
-  (each sample consumed once; ~10k samples over the run).
-- **Steps**: 10,000 (global batch size 1), LR 2e-5 cosine, seq len 4096,
-  Eagle3 TTT steps 3.
+  (each sample consumed once).
+- seq len 4096, Eagle3 TTT steps 3, global batch size 1.
 
 ## Validation (training-time, teacher-forced)
 
 Per-position draft accuracy on a fixed held-out val split, measured during
-training (teacher-forced argmax match; this is a *training-time* metric and is
-not directly comparable to runtime accept-length). `acc@i` is the accuracy at
-TTT position `i` (`i = 0, 1, 2`). `full_acc@i` requires positions `0..i` all
-correct; `cond_acc@i` is conditioned on `0..i-1` being correct. Continual FT
-lifts the weaker later positions most, while the already-strong position 0
-holds:
+training. `acc@i` is the accuracy at TTT position `i` (`i = 0, 1, 2`):
+`full_acc@i` requires positions `0..i` all correct; `cond_acc@i` is conditioned
+on `0..i-1` correct. This is a **training-time, teacher-forced** metric on a
+small online-sampled val split — it indicates per-position draft quality but is
+**not** a runtime accept-length and is not comparable across runs/splits.
 
-| metric | init | final | rel. |
-| --- | --- | --- | --- |
-| val_loss | 7.69 | 5.62 | -27% |
-| full_acc@0 | 0.614 | 0.656 | +7% |
-| full_acc@1 | 0.253 | 0.367 | +45% |
-| full_acc@2 | 0.085 | 0.184 | +116% |
-| cond_acc@1 | 0.412 | 0.559 | +36% |
-| cond_acc@2 | 0.336 | 0.499 | +49% |
+Best checkpoint of the refinement phase (this published checkpoint):
 
-(`cond_acc@0` equals `full_acc@0` by definition — position 0 has no prior to
-condition on — so it is the same 0.614 -> 0.656.)
+| metric | value |
+| --- | --- |
+| val_loss | 3.608 |
+| full_acc@0 | 0.799 |
+| full_acc@1 | 0.517 |
+| full_acc@2 | 0.306 |
+| cond_acc@1 | 0.648 |
+| cond_acc@2 | 0.594 |
 
-A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) is
-pending and will be added once measured.
+A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) on a
+common held-out set is pending and will be added once measured.
 
 ## Quick Start (vLLM >= 0.20.0)
 

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cc5e532f4dd84a3a3f58898a226ee8b1052f5cff4d542e02722d0cb7659c6df7
+oid sha256:e799138e5bccba32a9b02628e99ef93912999b0166ada4542fab797f73f3e28b
 size 6031210296