BobbieBieee/Osprey-Speculative-Decoding

Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Accepted at EMNLP 2026.

Paper: https://arxiv.org/abs/2609.09338
Code: https://github.com/LeanModels/Osprey

Adapted Osprey drafters for three target models. Every checkpoint starts from the same target-agnostic pretrained backbone (Qwen3-4B pruned to 2 layers, pretrained on FineWeb) and is adapted to its target with vocabulary alignment, zero-initialized QKV expansion, and on-policy EAGLE-3 distillation.

Path	Target	Adaptation data
qwen3-8b/osprey-chat	Qwen/Qwen3-8B	chat, 65k
qwen3-8b/osprey-code	Qwen/Qwen3-8B	code, 65k
qwen3-8b/osprey-commonsense	Qwen/Qwen3-8B	commonsense, 65k
qwen3-8b/osprey-finance	Qwen/Qwen3-8B	finance, 65k
qwen3-8b/osprey-math	Qwen/Qwen3-8B	math, 65k
llama33-70b/osprey	meta-llama/Llama-3.3-70B-Instruct	Open-PerfectBlend
minimax-m25/osprey	MiniMaxAI/MiniMax-M2.5	code, 70k

Each directory holds config.json and model.safetensors. Serving requires the SGLang patch shipped in the code repository (sglang_patches/), which adds multi-layer EAGLE-3 drafts and keeps the drafter's own embedding at load time.

Also in this repository:

Path	Contents
pretrained/qwen3-4b-2layer-fineweb-55k	The Stage-2 backbone every drafter above starts from: Qwen3-4B pruned to 2 layers and pretrained on FineWeb for 55k steps. Input to scripts/osprey/convert_checkpoint.py.
data/*_train_65k_qwen3_8B_4096.jsonl	The five Qwen3-8B adaptation splits (chat, code, commonsense, finance, math): 65k prompts each, responses regenerated by Qwen3-8B.
data/*_eval_512_qwen3_8B_4096.jsonl	The five 512-prompt evaluation splits behind Figure 2 and Table 1.
data/code_train_70k_minimax_m25_4096.jsonl	The MiniMax-M2.5 adaptation split: 70,897 coding prompts with responses regenerated by MiniMax-M2.5 (Tables 2–3).
data/code_eval_512_minimax_m25_4096.jsonl	Its 512-prompt held-out slice.
data/perfectblend_train_100k_llama33_70b.jsonl	The Llama-3.3-70B-Instruct adaptation split: 100,000 conversations from Open-PerfectBlend with responses regenerated by the target (Table 1).
data/perfectblend_eval_512_llama33_70b.jsonl	Its 512-prompt held-out slice.

The Llama-3.3-70B-Instruct splits are drawn from frankleeeee/PerfectBlend-Regenerated-Llama-3.3-70B-Instruct (1,419,775 conversations): records are numbered in file order, shuffled once with random.seed(42), and the first 100,000 / next 512 taken as train / eval.

The MiniMax-M2.5 splits take their prompts from the code subset of NVIDIA's Nemotron-Post-Training-Dataset-v2 (CC BY 4.0) and keep that dataset's metadata fields. The generator field is inherited from the source and does not describe the assistant turns here, which were regenerated by MiniMax-M2.5.

Citation

@inproceedings{bie2026osprey,
  title     = {Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding},
  author    = {Bie, Fengxiang and Jian, Yuqing and Yu, Yifan and Zhou, Zhongzhu and Shao, Zelei and Athiwaratkun, Ben and Song, Shuaiwen Leon and Xu, Chenfeng and Wu, Xiaoxia and Zhang, Tianyi},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026},
}