README.md

---
license: apache-2.0
library_name: pytorch
tags:
  - mixture-of-experts
  - deepseek-vl2
  - expert-prediction
  - prefetch
base_model: deepseek-ai/deepseek-vl2-tiny
---

# SpecPrefetch — DeepSeek-VL2-Tiny Future-Expert Predictor (single-horizon)

A lightweight predictor head trained on top of a frozen **DeepSeek-VL2-Tiny**
MoE model. It consumes the hidden state at an anchor layer `l` and predicts
the **Top-K expert set** that the *teacher router* will select **one layer
later** (`l+1`). The predictions are used at inference time to
**prefetch experts** before the router gets there, hiding cold-load latency
on GPUs with limited expert cache.

This checkpoint is a **single-horizon** (l+1 only) variant trained with the
fusion mode. (https://github.com/wei390/SpecPrefetch)

## Training spec

- Base model: `deepseek-ai/deepseek-vl2-tiny` (12 layers, 64 routed experts, top-6, layer 0 dense)
- Predictor: 2-layer residual MLP, hidden=1280, dropout=0.1
- Anchor layers: all 11 MoE layers (1..11)
- Horizon: 1 (predicts `l+1`)
- Fusion mode: `lora` (rank=64)
- Loss: KL between predictor distribution and teacher router softmax
- Frozen: backbone, experts, router, lm_head, embeddings

## Files

| File | Size | Purpose |
| --- | --- | --- |
| `model.safetensors` | 6.4 GB | Full model weights (frozen base + trained predictor + LoRA) |
| `config.json` | — | DeepSeek-VL2 config |
| `tokenizer.json` / `tokenizer_config.json` / `special_tokens_map.json` | — | Tokenizer |

## Reference numbers (full-set, next1)

| Dataset           | n     | recall@3 | recall@6 | recall@8 | exact@6 |
| ----------------- | ----- | -------- | -------- | -------- | ------- |
| ChartQA_TEST      | 2500  | 0.4963   | 0.8961   | 0.9683   | 0.4653  |
| OCRBench          | 1000  | 0.4936   | 0.8929   | 0.9645   | 0.4690  |
| HallusionBench    | 1129  | 0.4925   | 0.8872   | 0.9596   | 0.4488  |
| GSM8K             | 1319  | 0.4836   | 0.8435   | 0.9247   | 0.3282  |
| openai_humaneval  | 164   | 0.4761   | 0.8151   | 0.9065   | 0.2653  |

(Recall is averaged over `decode_tokens × anchor_layers` per sample. Random
baseline for recall@3 with 6/64 ground-truth experts ≈ 0.094.)

## Usage

Load the checkpoint into the [SpecPrefetch](https://github.com/wei390/SpecPrefetch) model:

```python
from huggingface_hub import snapshot_download
from model.configuration_deepseek_vl2 import DeepseekVL2DraftRouterConfig
from model.modeling_deepseek_vl2 import (
    DeepseekVL2DraftRouterForConditionalGeneration,
    load_pretrained_weights,
)

ckpt = snapshot_download("jinwei001/SpecPrefetch_deepseekvl2")
config = DeepseekVL2DraftRouterConfig.from_pretrained(
    ckpt,
    future_expert_predictor_enabled=True,
    future_expert_fusion_mode="lora",
)
model = DeepseekVL2DraftRouterForConditionalGeneration(config)
load_pretrained_weights(model, ckpt)
```