BCCard/MoAI-gemma-4-31B-it-speculator.dspark

gemma-4-31B-it DSpark Draft (Korean / Finance-optimized)

Repo id assumed as BCCard/MoAI-gemma-4-31B-it-speculator.dspark (parallel to the ...speculator.eagle3 siblings). Adjust the model field in the serving command if you publish under a different id.

A DSpark draft (speculator) that accelerates generation from the dense gemma-4-31B-it verifier via speculative decoding. DSpark pairs a DFlash-style parallel backbone with a sequential Markov head and a confidence scheduler, so the draft proposes a variable number of tokens per step and stops early once its own confidence drops. This checkpoint was trained from scratch — no public DSpark checkpoint exists for gemma-4-31B — on 600K on-policy prompts whose answers were regenerated by the verifier itself.

• Method: DSpark — Confidence-Scheduled Speculative Decoding (DeepSeek-AI, ICML 2026), trained with vllm-project/speculators
• Verifier (target): BCCard/gemma-4-31B-it-FP8-Dynamic (31B dense, FP8; used for serving and hidden-state extraction)
• Reused weights: BF16 embed / lm_head from google/gemma-4-31B-it (standard draft init)
• Draft recipe: 3 draft layers (--num-layers 3), DSpark loss {"ce": 0.1, "tv": 0.9}, block-size 8, Markov head (rank 256, vanilla) + confidence head (with-markov, α = 1.0)
• Training data: 600K on-policy prompts (see below)
• Sequence length: 8192 (training); the draft serves at any length up to the verifier's context
• Training: 5 epochs, lr 1e-4 (cosine → ~0), draft vocab 32000
• speculators pin: 2d97b17b (main, DSpark online-training support)

Training data (600K, on-policy)

All responses were regenerated on-policy by the verifier (BCCard/gemma-4-31B-it-FP8-Dynamic); only the source prompts were reused. Publicly available drafts are English-first and accept Korean poorly, so this draft is retrained on a Korean/English/finance mix.

Source	Prompts	Language
Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered	~300K	English
BCCard/BCAI-Finance-Kor-1862K	~150K	Korean (finance)
BCCard/gemma-4-31B-korean-on-policy-150k	~150K	Korean
Total	~600K

Serving (vLLM)

DSpark serving requires a vLLM main build (PR #46995, merged 2026-07-01) — it is not in v0.24.0.

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve BCCard/gemma-4-31B-it-FP8-Dynamic -tp 1 \
  --max-model-len 8192 \
  --speculative-config '{
    "model": "BCCard/MoAI-gemma-4-31B-it-speculator.dspark",
    "method": "dspark",
    "num_speculative_tokens": 7,
    "draft_tensor_parallel_size": 1,
    "draft_sample_method": "probabilistic"
  }'

• The draft shares the verifier's tokenizer.
• num_speculative_tokens is an upper bound: DSpark's confidence scheduler proposes fewer when uncertain, so a larger ceiling costs little. Given the ~2.2 mean accepted length, 3–8 is a reasonable tuning range for your traffic.
• draft_sample_method: "probabilistic" (default) or "argmax".
• On Blackwell (sm_120 / sm_121) keep VLLM_USE_FLASHINFER_SAMPLER=0 and unset PYTORCH_CUDA_ALLOC_CONF.

Performance (validation, best checkpoint)

Measured at the end of training (epoch 5, checkpoint_best) on the held-out validation split against the BCCard/gemma-4-31B-it-FP8-Dynamic verifier.

• Mean accepted length ≈ 2.22 tokens/step → roughly ~2.2× fewer verifier forward passes in the ideal case (measure wall-clock TPS on your own traffic).
• Accept rate 0.266 · full_acc 0.317.

Per-position draft accuracy (teacher-forced):

position	1	2	3	4	5	6	7
acc	0.610	0.413	0.318	0.264	0.228	0.201	0.179

The confidence head (which drives the dynamic draft length) is well-calibrated — mean absolute error 0.133, cumulative-product bias −0.005 — so its early-stopping decisions are reliable. Training-time accepted length was ≈ 2.88; validation loss was best at the final epoch (no divergence), so the train/val gap reflects validation-set difficulty rather than overfitting.

Limitations

• Optimized for Korean, English, and Korean-finance QA. Very different domains (code, long-form reasoning) may benefit from one more on-policy cycle on domain-matched prompts, which typically raises acceptance.
• Acceptance is tied to the specific verifier BCCard/gemma-4-31B-it-FP8-Dynamic. Pairing the draft with a different target (or a non--it base) will change results.
• Requires a vLLM main build with DSpark support. Known upstream constraints include no PP>1 MTP (#46994), a TP>1 + batching deadlock (#41404), and a prefix-cache accuracy regression (#43559) — verify current vLLM issues before production.
• Trained from scratch (no warm start), so acceptance may still improve with more data/epochs.

License

Apache 2.0, consistent with the base Gemma 4 (google/gemma-4-31B-it, Apache 2.0) and the verifier BCCard/gemma-4-31B-it-FP8-Dynamic. Apache 2.0 permits commercial use, modification, and redistribution with attribution and disclosure of modifications. (Informational, not legal advice.)