sionic-ai/Solar-Open2-250B-ultrafast-Draft-LM-head

Solar-Open2 Speculative Draft (500K, 2 epochs)

Block-draft speculative-decoding draft model for Upstage Solar Open2 250B-A15B (48 layers = 12 NoPE GQA + 36 linear-attention layers, 320-expert MoE top-8). The draft proposes a block of γ tokens in a single forward pass, conditioned on five taps of the target model's hidden states (layers 3 / 15 / 27 / 39 / 47), and the target verifies the whole block in one step.

This is the 500K-scale, 2-epoch checkpoint (our best to date). It is not a standalone language model — it only functions as a draft attached to the Solar Open2 250B target inside a serving stack with block-draft speculative decoding support for this draft format.

Architecture

Component	Spec
Draft core (trainable)	5 transformer layers, hidden 4096, GQA 32/8 heads, sliding-window 128 — 0.90B params
Hidden-state conditioning	5-tap fc (5×4096 → 4096), taps at target layers [3, 15, 27, 39, 47]
Draft head	rank-256 factorized (vocab 196,608)
Confidence head	enabled (per-position acceptance calibration)
Embedding / LM head	copied frozen from the target (805M each) — total checkpoint 2.51B
Block size	up to 7 mask tokens (γ ≤ 7, mask token id 26)

Training

• Data: 500K-scale multi-domain prompt collection — 415K-prompt manifest after global dedup and long-context caps (code, math, structured output, safety, multilingual, long-context up to 128K). ~1.2B tokens were pushed through the target model on 8× B300 for hidden-state distillation, including ~540M freshly generated response tokens.
• Recipe: 2 epochs, streaming producer/trainer pipeline (target forward and draft training co-located on 8 GPUs), chunked KL loss (bit-identical forward), 2048 stratified anchors per batch, lr 6e-4, total-seq-len 8192.
• Epoch 2 ran with a 95/5 train/holdout split (holdout-tail validation, never trained).

Results

Metric	Previous draft (150K)	This model
val accept_len (γ=7)	2.486	3.431 (+38%)
Serving accept_len γ=3 / γ=7	2.27 / —	2.75 / 3.54
gsm8k (5-shot greedy, spec on)	0.935	0.930–0.940 (lossless band)

On our optimized serving branch (B300, TP8, fp8 target, batch-1 serial decode), this draft lifted end-to-end throughput from 335 tok/s (previous draft) to 474 tok/s at γ=7 together with the accompanying kernel/scheduler rounds. Acceptance is the dominant factor: at equal server config the draft alone is worth roughly +8% TPS per +10% accept_len.

Operating notes

• γ (block size) 4–7 are all near-optimal with this draft; γ=7 maximizes accept.
• Keep the draft unquantized: fp8-quantizing the draft costs ~2% accept for negligible speed.
• The confidence head supports adaptive per-request verify budgets when paired with a step-cost table and temperature calibration.

Limitations

• Not usable standalone; requires the Solar Open2 250B target and a serving build with block-draft speculative decoding support for this format.
• Trained mostly on ≤32K contexts; long-context (32K–128K) acceptance is weaker.
• License/usage follows the base model (Upstage Solar Open2) — this checkpoint contains the target's frozen embedding/LM-head weights.