File size: 2,855 Bytes
a708d92 3ba5245 a708d92 3ba5245 3d742a1 3ba5245 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
license: apache-2.0
library_name: pytorch
tags:
- mixture-of-experts
- deepseek-vl2
- expert-prediction
- prefetch
base_model: deepseek-ai/deepseek-vl2-tiny
---
# SpecPrefetch — DeepSeek-VL2-Tiny Future-Expert Predictor (single-horizon)
A lightweight predictor head trained on top of a frozen **DeepSeek-VL2-Tiny**
MoE model. It consumes the hidden state at an anchor layer `l` and predicts
the **Top-K expert set** that the *teacher router* will select **one layer
later** (`l+1`). The predictions are used at inference time to
**prefetch experts** before the router gets there, hiding cold-load latency
on GPUs with limited expert cache.
This checkpoint is a **single-horizon** (l+1 only) variant trained with the
fusion mode. (https://github.com/wei390/SpecPrefetch)
## Training spec
- Base model: `deepseek-ai/deepseek-vl2-tiny` (12 layers, 64 routed experts, top-6, layer 0 dense)
- Predictor: 2-layer residual MLP, hidden=1280, dropout=0.1
- Anchor layers: all 11 MoE layers (1..11)
- Horizon: 1 (predicts `l+1`)
- Fusion mode: `lora` (rank=64)
- Loss: KL between predictor distribution and teacher router softmax
- Frozen: backbone, experts, router, lm_head, embeddings
## Files
| File | Size | Purpose |
| --- | --- | --- |
| `model.safetensors` | 6.4 GB | Full model weights (frozen base + trained predictor + LoRA) |
| `config.json` | — | DeepSeek-VL2 config |
| `tokenizer.json` / `tokenizer_config.json` / `special_tokens_map.json` | — | Tokenizer |
## Reference numbers (full-set, next1)
| Dataset | n | recall@3 | recall@6 | recall@8 | exact@6 |
| ----------------- | ----- | -------- | -------- | -------- | ------- |
| ChartQA_TEST | 2500 | 0.4963 | 0.8961 | 0.9683 | 0.4653 |
| OCRBench | 1000 | 0.4936 | 0.8929 | 0.9645 | 0.4690 |
| HallusionBench | 1129 | 0.4925 | 0.8872 | 0.9596 | 0.4488 |
| GSM8K | 1319 | 0.4836 | 0.8435 | 0.9247 | 0.3282 |
| openai_humaneval | 164 | 0.4761 | 0.8151 | 0.9065 | 0.2653 |
(Recall is averaged over `decode_tokens × anchor_layers` per sample. Random
baseline for recall@3 with 6/64 ground-truth experts ≈ 0.094.)
## Usage
Load the checkpoint into the [SpecPrefetch](https://github.com/wei390/SpecPrefetch) model:
```python
from huggingface_hub import snapshot_download
from model.configuration_deepseek_vl2 import DeepseekVL2DraftRouterConfig
from model.modeling_deepseek_vl2 import (
DeepseekVL2DraftRouterForConditionalGeneration,
load_pretrained_weights,
)
ckpt = snapshot_download("jinwei001/SpecPrefetch_deepseekvl2")
config = DeepseekVL2DraftRouterConfig.from_pretrained(
ckpt,
future_expert_predictor_enabled=True,
future_expert_fusion_mode="lora",
)
model = DeepseekVL2DraftRouterForConditionalGeneration(config)
load_pretrained_weights(model, ckpt)
```
|