Cornmonster commited on
Commit
043833a
·
verified ·
1 Parent(s): 837335c

README: note input embedding is loaded from verifier (not shipped), checkpoint ~3.7GB

Browse files
Files changed (1) hide show
  1. README.md +3 -0
README.md CHANGED
@@ -35,6 +35,9 @@ Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.7-
35
  - **Algorithm**: EAGLE-3 with MLA, single draft decoder layer.
36
  - **Verifier**: Kimi-K2.7-Code. The draft reuses the verifier's frozen embedding / lm_head / norm
37
  and trains one MLA decoder layer plus an auxiliary-hidden-state fusion layer.
 
 
 
38
  - **Draft vocabulary**: full 163,840-token vocabulary (no truncation).
39
 
40
  ### Training Setup
 
35
  - **Algorithm**: EAGLE-3 with MLA, single draft decoder layer.
36
  - **Verifier**: Kimi-K2.7-Code. The draft reuses the verifier's frozen embedding / lm_head / norm
37
  and trains one MLA decoder layer plus an auxiliary-hidden-state fusion layer.
38
+ - **Shared input embedding**: the input embedding is **not shipped** in this checkpoint — it is
39
+ loaded from the verifier at inference (vLLM's `Eagle3DeepseekV2ForCausalLM` shares the target's
40
+ `embed_tokens` when the draft weights omit it). This keeps the checkpoint compact (~3.7 GB).
41
  - **Draft vocabulary**: full 163,840-token vocabulary (no truncation).
42
 
43
  ### Training Setup