metadata
license: apache-2.0
library_name: pytorch
tags:
- mixture-of-experts
- deepseek-vl2
- expert-prediction
- prefetch
base_model: deepseek-ai/deepseek-vl2-tiny
SpecPrefetch — DeepSeek-VL2-Tiny Future-Expert Predictor (single-horizon)
A lightweight predictor head trained on top of a frozen DeepSeek-VL2-Tiny
MoE model. It consumes the hidden state at an anchor layer l and predicts
the Top-K expert set that the teacher router will select one layer
later (l+1). The predictions are used at inference time to
prefetch experts before the router gets there, hiding cold-load latency
on GPUs with limited expert cache.
This checkpoint is a single-horizon (l+1 only) variant trained with the fusion mode. (https://github.com/wei390/SpecPrefetch)
Training spec
- Base model:
deepseek-ai/deepseek-vl2-tiny(12 layers, 64 routed experts, top-6, layer 0 dense) - Predictor: 2-layer residual MLP, hidden=1280, dropout=0.1
- Anchor layers: all 11 MoE layers (1..11)
- Horizon: 1 (predicts
l+1) - Fusion mode:
lora(rank=64) - Loss: KL between predictor distribution and teacher router softmax
- Frozen: backbone, experts, router, lm_head, embeddings
Files
| File | Size | Purpose |
|---|---|---|
model.safetensors |
6.4 GB | Full model weights (frozen base + trained predictor + LoRA) |
config.json |
— | DeepSeek-VL2 config |
tokenizer.json / tokenizer_config.json / special_tokens_map.json |
— | Tokenizer |
Reference numbers (full-set, next1)
| Dataset | n | recall@3 | recall@6 | recall@8 | exact@6 |
|---|---|---|---|---|---|
| ChartQA_TEST | 2500 | 0.4963 | 0.8961 | 0.9683 | 0.4653 |
| OCRBench | 1000 | 0.4936 | 0.8929 | 0.9645 | 0.4690 |
| HallusionBench | 1129 | 0.4925 | 0.8872 | 0.9596 | 0.4488 |
| GSM8K | 1319 | 0.4836 | 0.8435 | 0.9247 | 0.3282 |
| openai_humaneval | 164 | 0.4761 | 0.8151 | 0.9065 | 0.2653 |
(Recall is averaged over decode_tokens × anchor_layers per sample. Random
baseline for recall@3 with 6/64 ground-truth experts ≈ 0.094.)
Usage
Load the checkpoint into the SpecPrefetch model:
from huggingface_hub import snapshot_download
from model.configuration_deepseek_vl2 import DeepseekVL2DraftRouterConfig
from model.modeling_deepseek_vl2 import (
DeepseekVL2DraftRouterForConditionalGeneration,
load_pretrained_weights,
)
ckpt = snapshot_download("jinwei001/SpecPrefetch_deepseekvl2")
config = DeepseekVL2DraftRouterConfig.from_pretrained(
ckpt,
future_expert_predictor_enabled=True,
future_expert_fusion_mode="lora",
)
model = DeepseekVL2DraftRouterForConditionalGeneration(config)
load_pretrained_weights(model, ckpt)