Super-squash branch 'main' using huggingface_hub

d55cc99 8 days ago

1.11 kB

license: other
base_model: moonshotai/Kimi-K2.7-Code
library_name: transformers
tags:
  - speculative-decoding
  - eagle3
  - eagle
  - draft-model
  - kimi-k2
  - specforge

Kimi-K2.7-Code EAGLE3 draft

EAGLE3 speculative-decoding draft model for moonshotai/Kimi-K2.7-Code, trained with SpecForge (PR #593) on NVIDIA Nemotron-Post-Training-Dataset-v2 (stem+chat+math+code).

Single-layer Llama-style EAGLE3 draft (LlamaForCausalLMEagle3, hidden 7168); consumes target aux hidden states at layers [1,29,57].
Target vocab/tokenizer: Kimi-K2.7-Code (vocab 163840); reduced draft vocab 32000 (t2d/d2t mapping, 97.5% token coverage on the training mix).
Checkpoint: epoch_1_step_98000 — Work-in-progress snapshot (epoch_1_step_98000) — training still running.

Intended as the EAGLE3 draft in SGLang speculative decoding paired with the Kimi-K2.7-Code target. Trained as an A/B counterpart to the DFlash draft (cm00cm/Kimi-K2.7-Code-DFlash).