v3 fine-tune: L2-SP anchor + ttt=4 off-policy (accept_len 2.345, pos 0.648/0.419/0.278)

README.md

@@ -1,68 +1,54 @@
 ---
 license: other
-base_model: lightseekorg/kimi-k2.6-eagle3-mla
+base_model: moonshotai/Kimi-K2.6
 tags:
 - text-generation
 - speculative-decoding
 - eagle3
 - kimi-k2.6
 - mla
-- camelot
+- torchspec
 ---
 
-# kimi-k2.6-eagle3-mla (continual fine-tune)
+# kimi-k2.6-eagle3-mla
 
 Eagle3 MTP draft model with MLA (Multi-Latent Attention) for accelerating
 inference of [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6).
 
-This checkpoint is a **continual fine-tune** of
-[lightseekorg/kimi-k2.6-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3-mla),
-further trained on an in-house instruction distribution. The architecture,
-config, and tensor layout are identical to the base model, so it is a drop-in
-replacement in the same vLLM serving path.
+This is a fine-tuned draft, anchored to the official
+[lightseekorg/kimi-k2.6-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3-mla)
+initialization. It targets multi-hop (downstream-position) acceptance while
+preserving the first-hop gain, evaluated by runtime accept-length on a frozen
+full-context held-out set.
 
-## Training Setup
+## Fine-tune setup
 
-- **Init**: continual FT from `lightseekorg/kimi-k2.6-eagle3-mla`.
-- **Framework**: Camelot online speculative-decoding training — concurrent
-  FSDP training + vLLM rollout with cross-node hidden-state transfer over
-  Mooncake (RDMA).
-- **Topology**: 1 datagen node (Kimi-K2.6, TP=8) -> Mooncake -> 1 trainer GPU.
-- **Schedule**: an initial 10k-step cosine phase (LR 2e-5), then a constant
-  low-LR (2e-6) refinement phase continued from the best checkpoint. This
-  published checkpoint is the best-val checkpoint of the refinement phase.
-- **Data**: online-generated hidden states from a ~100k-prompt instruction set
-  (each sample consumed once).
-- seq len 4096, Eagle3 TTT steps 3, global batch size 1.
+- **Init**: lightseekorg/kimi-k2.6-eagle3-mla (official MLA weights)
+- **Objective**: Eagle3 distillation + multi-step TTT supervision
+  (`ttt_steps=4`, `ttt_step_loss_decay=1.0`, off-policy downstream tokens)
+- **Anti-over-specialization**: L2-SP weight-space anchor toward the init
+  (penalize trainable-param drift; lambda=1e-4)
+- **Optimizer**: lr 2e-5, cosine schedule
+- **Checkpoint**: best by held-out runtime accept-length
 
-## Validation (training-time, teacher-forced)
+## Performance
 
-Per-position draft accuracy on a fixed held-out val split, measured during
-training. `acc@i` is the accuracy at TTT position `i` (`i = 0, 1, 2`):
-`full_acc@i` requires positions `0..i` all correct; `cond_acc@i` is conditioned
-on `0..i-1` correct. This is a **training-time, teacher-forced** metric on a
-small online-sampled val split — it indicates per-position draft quality but is
-**not** a runtime accept-length and is not comparable across runs/splits.
+Primary metric is **accept_length** — average tokens accepted per speculation
+step with `num_speculative_tokens=3` (higher is better). Per-position numbers
+are conditional acceptance rates at hop 0/1/2. Evaluated on a frozen
+full-context held-out judge set (912 prompts, greedy), vLLM 0.20.0, 8x H200,
+TP=8, max-model-len 32768.
 
-Best checkpoint of the refinement phase (this published checkpoint):
+| Model | accept_len | pos-0 | pos-1 | pos-2 |
+|-------|-----------:|------:|------:|------:|
+| lightseek (official init) | 2.30 | 0.633 | 0.404 | 0.264 |
+| this model | **2.345** | **0.648** | **0.419** | **0.278** |
 
-| metric | value |
-| --- | --- |
-| val_loss | 3.608 |
-| full_acc@0 | 0.799 |
-| full_acc@1 | 0.517 |
-| full_acc@2 | 0.306 |
-| cond_acc@1 | 0.648 |
-| cond_acc@2 | 0.594 |
+This draft improves first-hop acceptance over the official init while also
+lifting the downstream positions (pos-1, pos-2), yielding a higher overall
+accept length.
 
-A runtime `accept_length` benchmark (vLLM 0.20, `num_speculative_tokens=3`) on a
-common held-out set is pending and will be added once measured.
+## Usage
 
-## Quick Start (vLLM >= 0.20.0)
-
-```bash
-vllm serve moonshotai/Kimi-K2.6 \
-    --tensor-parallel-size 8 \
-    --speculative-config '{"model": "k-l-lambda/kimi-k2.6-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
-    --trust-remote-code
-```
+Serve with vLLM as the speculative draft for Kimi-K2.6, with
+`num_speculative_tokens=3` in the speculative-config.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e799138e5bccba32a9b02628e99ef93912999b0166ada4542fab797f73f3e28b
+oid sha256:57cf1a3b1cd6153619f37a42410e6123fd899f15d2f95d230dafe68399825d04
 size 6031210296