jinwei001/SpecPrefetch_deepseekvl2

SpecPrefetch — DeepSeek-VL2-Tiny Future-Expert Predictor (single-horizon)

A lightweight predictor head trained on top of a frozen DeepSeek-VL2-Tiny MoE model. It consumes the hidden state at an anchor layer l and predicts the Top-K expert set that the teacher router will select one layer later (l+1). The predictions are used at inference time to prefetch experts before the router gets there, hiding cold-load latency on GPUs with limited expert cache.

This checkpoint is a single-horizon (l+1 only) variant trained with the fusion mode. (https://github.com/wei390/SpecPrefetch)

Training spec

• Base model: deepseek-ai/deepseek-vl2-tiny (12 layers, 64 routed experts, top-6, layer 0 dense)
• Predictor: 2-layer residual MLP, hidden=1280, dropout=0.1
• Anchor layers: all 11 MoE layers (1..11)
• Horizon: 1 (predicts l+1)
• Fusion mode: lora (rank=64)
• Loss: KL between predictor distribution and teacher router softmax
• Frozen: backbone, experts, router, lm_head, embeddings

Files

File	Size	Purpose
model.safetensors	6.4 GB	Full model weights (frozen base + trained predictor + LoRA)
config.json	—	DeepSeek-VL2 config
tokenizer.json / tokenizer_config.json / special_tokens_map.json	—	Tokenizer

Reference numbers (full-set, next1)

Dataset	n	recall@3	recall@6	recall@8	exact@6
ChartQA_TEST	2500	0.4963	0.8961	0.9683	0.4653
OCRBench	1000	0.4936	0.8929	0.9645	0.4690
HallusionBench	1129	0.4925	0.8872	0.9596	0.4488
GSM8K	1319	0.4836	0.8435	0.9247	0.3282
openai_humaneval	164	0.4761	0.8151	0.9065	0.2653

(Recall is averaged over decode_tokens × anchor_layers per sample. Random baseline for recall@3 with 6/64 ground-truth experts ≈ 0.094.)

Usage

Load the checkpoint into the SpecPrefetch model:

from huggingface_hub import snapshot_download
from model.configuration_deepseek_vl2 import DeepseekVL2DraftRouterConfig
from model.modeling_deepseek_vl2 import (
    DeepseekVL2DraftRouterForConditionalGeneration,
    load_pretrained_weights,
)

ckpt = snapshot_download("jinwei001/SpecPrefetch_deepseekvl2")
config = DeepseekVL2DraftRouterConfig.from_pretrained(
    ckpt,
    future_expert_predictor_enabled=True,
    future_expert_fusion_mode="lora",
)
model = DeepseekVL2DraftRouterForConditionalGeneration(config)
load_pretrained_weights(model, ckpt)