Flow-Drafter-9B-v2

Expanded-data revision of Flow-Drafter-9B — a speculative-decoding draft model for Qwen/Qwen3.5-9B (Chained-Flow, joint-VAE drafter).

What changed in v2: trained on a broader, diversity-weighted mix (~46k rows, 10 sources) adding multi-turn chat (UltraChat), diverse instructions (No-Robots), creative prose (WritingPrompts), summarization (CNN/DailyMail) and translation (OPUS) on top of the original math/code/STEM core, to lift acceptance on the low-predictability domains.

Lossless by construction: a proposed token is emitted only if the target itself would have emitted it. Outputs can still differ from unspeculated decode, because verifying K+1 positions at once changes the target's own fp16 reduction order — the observed divergence is comparable to the run-to-run difference between two unspeculated runs.

Usage

pip install chained-flow

CF_DRAFTER_DIR=selimaktas/Flow-Drafter-9B-v2 \
vllm serve Qwen/Qwen3.5-9B --async-scheduling \
  --speculative-config '{"method":"custom_class","model":"chained_flow.vllm_plugin.flow_proposer.FlowDrafterProposer","num_speculative_tokens":5}'

Measured under vLLM — acceptance and speedup, per domain

Condition: batch 1 (concurrency 1), chain proposer at K=5, greedy, fixed 256 output tokens per request (ignore_eos) so both arms do identical work, async scheduling on for every arm including the baseline, 3 repeats (pooled spread ≤0.1%). Benchmark: RedHatAI/speculator_benchmarks, 8 distinct domains × 25 prompts, one run per domain, never pooled into a single split. Harness: guidellm 0.6.0 driving vllm serve (vLLM 0.25.1). acceptance = 1 + num_accepted_tokens / num_drafts from vLLM's own Prometheus counters (bonus token included, so a non-speculative baseline is 1.00). speedup = chain tok/s ÷ base tok/s, prefill included in the denominator.

domain	acceptance (tokens/step)	speedup vs base
HumanEval (code)	2.38	1.61x
math_reasoning	2.85	1.92x
qa (short free-form)	1.96	1.34x
question (MT-bench)	2.05	1.40x
rag	2.10	1.42x
summarization	2.00	1.35x
tool_call	2.16	1.45x
translation (de->en)	1.49	1.02x
POOLED (all 8 domains)	2.06	1.40x

Quote the pooled figure, not the best domain. The spread is wide — 1.92x on math_reasoning down to 1.02x on translation, which is bare parity: at that acceptance the draft pass costs about what it saves. The pooled 1.40x already includes it.

At 9B the base decode step is more expensive than at 4B, so the drafter has more headroom to buy back, and every domain is at or above parity. The 27B drafter returns more still.

CF_SPEC_MAX_BATCH ships on by default and resolves to a decode batch of 4 at this size, holding a loaded server near parity rather than below it.

Files

• model.safetensors — the drafter (flow experts, Markov head, path head, jointly-trained VAE)
• chained_flow_tree_config.json — architecture + loss config
• vae/ — the base VAE checkpoint

Drafts all K future hidden states in one flow pass and the base model verifies them. Trained 2-GPU DDP.

Companion models: Flow-Drafter-4B-v2, Flow-Drafter-Qwen3.5-27B-v2.