README: note input embedding is loaded from verifier (not shipped), checkpoint ~3.7GB
Browse files
README.md
CHANGED
|
@@ -35,6 +35,9 @@ Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.7-
|
|
| 35 |
- **Algorithm**: EAGLE-3 with MLA, single draft decoder layer.
|
| 36 |
- **Verifier**: Kimi-K2.7-Code. The draft reuses the verifier's frozen embedding / lm_head / norm
|
| 37 |
and trains one MLA decoder layer plus an auxiliary-hidden-state fusion layer.
|
|
|
|
|
|
|
|
|
|
| 38 |
- **Draft vocabulary**: full 163,840-token vocabulary (no truncation).
|
| 39 |
|
| 40 |
### Training Setup
|
|
|
|
| 35 |
- **Algorithm**: EAGLE-3 with MLA, single draft decoder layer.
|
| 36 |
- **Verifier**: Kimi-K2.7-Code. The draft reuses the verifier's frozen embedding / lm_head / norm
|
| 37 |
and trains one MLA decoder layer plus an auxiliary-hidden-state fusion layer.
|
| 38 |
+
- **Shared input embedding**: the input embedding is **not shipped** in this checkpoint — it is
|
| 39 |
+
loaded from the verifier at inference (vLLM's `Eagle3DeepseekV2ForCausalLM` shares the target's
|
| 40 |
+
`embed_tokens` when the draft weights omit it). This keeps the checkpoint compact (~3.7 GB).
|
| 41 |
- **Draft vocabulary**: full 163,840-token vocabulary (no truncation).
|
| 42 |
|
| 43 |
### Training Setup
|