alice1001/dspark_qwen3_4b_block7

Qwen3-4B DSpark Block-7 — Open-PerfectBlend Regen, Step 18,000

This repository contains the step 18,000 DSpark draft checkpoint trained for speculative decoding with Qwen/Qwen3-4B as the target model. It is not a standalone chat or text-generation model. It must be loaded with the DeepSpec Qwen3DSparkModel implementation and used together with the Qwen3-4B target model.

The published model.safetensors is the checkpoint's original BF16 file, copied byte-for-byte. It was not re-encoded, and optimizer, RNG, and 38 per-rank training-state files are not included. Results for later checkpoints in this card document convergence of the same training run; this repository still contains step 18,000 weights only.

Model architecture

Item	Value
Draft architecture	Qwen3DSparkModel
Target model	Qwen/Qwen3-4B
DSpark block size	7
Draft transformer layers	5
Target feature layers	[1, 9, 17, 25, 33]
Hidden size	2,560
Anchors	512
Markov head	vanilla, rank 256
Confidence head	Enabled; Markov features enabled
Weight dtype	BF16
Maximum position configuration	40,960

The legacy internal experiment name contains dspark_block8, but the saved checkpoint configuration and actual model use block_size=7. The repository name follows the authoritative saved configuration.

Training data

The model was trained with DeepSpec's Qwen3DSparkTrainer on 1,339,649 Open-PerfectBlend conversations. Answers were regenerated by Qwen3-4B with thinking disabled. The public prompt/answer dataset is alice1001/open-perfectblend-qwen3-4b-regen.

Training target caches were generated from each complete prompt + regenerated answer sequence. The unified cache contained 628 shards and retained at most 4,090 tokens per sample. Target hidden states came from Qwen3-4B layers [1, 9, 17, 25, 33] in BF16.

Training configuration and duration

Setting	Value
Trainer	Qwen3DSparkTrainer
Hardware	38 × NVIDIA A800 GPUs
Distributed layout	19 logical workers × 2 GPUs; actual GPU world size 38
Precision	BF16
Local batch size	1 per GPU
Global batch size	494
Gradient accumulation	13
Epochs / optimizer steps	10 / 27,110
Published checkpoint	step_18000（66.40% of the run; approximately 6.64 epochs）
Training sequence length	4,096（target-cache content up to 4,090）
Draft target layers	[1, 9, 17, 25, 33]
Draft layers / block size	5 / 7
Anchors / Markov rank	512 / 256
Confidence-head alpha	1.0
Loss weights	CE 0.1, L1 0.9, decay gamma 4.0
Peak learning rate	Approximately 6e-4
LR schedule	4% warmup, then cosine decay to zero
Weight decay	0.0
Maximum gradient norm	1.0
Sharding strategy	no_shard
Torch compile	Enabled
Logging / checkpoint interval	10 / 3,000 optimizer steps
Data workers	1
Full-run observed wall time	191.97 hours, from the first exported TensorBoard point at step=10 to the final point at step=27110
Time to published checkpoint	123.57 hours, from the same TensorBoard reference to step=18000

The wall times above are TensorBoard-observed intervals. They include any wall-clock pauses during the run but exclude setup time before the first exported point and the first nine optimizer steps.

At the published checkpoint, the recorded training metrics were loss=0.5449, accept_ratio@0=0.9180, and learning_rate=0.00016383. The full run completed at step 27,110 with loss=0.4610, accept_ratio@0=0.9270, and learning rate zero.

Training curves

The following plots were exported directly from TensorBoard. The horizontal axis is optimizer step. TensorBoard's train/accept_rate@0 is labeled accept_ratio@0 here to match the training report.

Training loss

Training accept ratio at proposal position 0

Learning rate

Loss decreased from 3.5494 at step 10 to 0.4610 at step 27,110; accept_ratio@0 increased from 0.0001 to 0.9270. After step 3,000, training loss mainly fluctuated between approximately 0.45 and 0.8.

Checkpoint progression

All checkpoints below were evaluated with the same DeepSpec runtime settings: max_new_tokens=2048, temperature=1.0, confidence_threshold=0.0, and seed=980406. The nine tasks are GSM8K, MATH-500, AIME 2025, MBPP, HumanEval, LiveCodeBench, MT-Bench, Alpaca, and Arena-Hard-v2.

These are speculative-decoding runtime metrics, not task answer accuracy or pass rate. accept_len is the average accepted length $\tau$; higher is better. verify_rate is the accepted-token fraction for this evaluation configuration.

Checkpoint	Train loss	Train accept ratio@0	LR	9-task mean accept_len	9-task mean verify_rate	Arena-Hard accept_len	Arena-Hard verify_rate
step_3000	0.7522	0.8707	0.00059201	4.0989	0.5145	2.68	0.3356
step_6000	0.6990	0.8900	0.00054871	4.3544	0.5465	2.93	0.3667
step_9000	0.6489	0.8988	0.00047315	4.5100	0.5662	3.07	0.3847
step_12000	0.5594	0.9064	0.00037513	4.6156	0.5795	3.18	0.3983
step_15000	0.5164	0.9116	0.00026736	4.6656	0.5857	3.21	0.4013
step_18000（published）	0.5449	0.9180	0.00016383	4.7378	0.5945	3.29	0.4118
step_21000	0.5015	0.9220	0.00007796	4.7422	0.5954	3.32	0.4160
step_24000	0.4896	0.9272	0.00002089	4.7411	0.5952	3.29	0.4121
step_27000	0.5341	0.9278	0.00000003	4.7444	0.5955	3.32	0.4154
step_27110	0.4610	0.9270	0.00000000	4.7667	0.5983	3.33	0.4170

Paper, official checkpoint, and this run

The comparison follows the task order and accepted-length definition used in the DSpark paper's Table 1. “Official checkpoint reproduction” is a local reproduction of the released Qwen3-4B DSpark checkpoint under the same fixed evaluation protocol. The final columns compare each nine-task mean with the paper mean of 4.7267:

$${\mathrm{\Delta }}_{\mathrm{\%}}=\frac{{\tau }_{\mathrm{c}\mathrm{h}\mathrm{e}\mathrm{c}\mathrm{k}\mathrm{p}\mathrm{o}\mathrm{i}\mathrm{n}\mathrm{t}\text{-}\mathrm{a}\mathrm{v}\mathrm{g}}-{\tau }_{\mathrm{p}\mathrm{a}\mathrm{p}\mathrm{e}\mathrm{r}\text{-}\mathrm{a}\mathrm{v}\mathrm{g}}}{{\tau }_{\mathrm{p}\mathrm{a}\mathrm{p}\mathrm{e}\mathrm{r}\text{-}\mathrm{a}\mathrm{v}\mathrm{g}}}\times 100\mathrm{\%}\mathrm{.}\backslash Delta\_\{\backslash \%\}\; =\; \backslash frac\{\backslash tau\_\{\backslash mathrm\{checkpoint\backslash text\{-\}avg\}\}\; -\; \backslash tau\_\{\backslash mathrm\{paper\backslash text\{-\}avg\}\}\}\{\backslash tau\_\{\backslash mathrm\{paper\backslash text\{-\}avg\}\}\}\; \backslash times\; 100\backslash \%.$$

Source	GSM8K	MATH500	AIME25	MBPP	HumanEval	LCB	MT-Bench	Alpaca	Arena-Hard	9-task mean	vs. paper	vs. paper %
DSpark paper, Qwen3-4B	6.11	5.70	4.89	5.13	5.38	4.86	3.64	3.54	3.29	4.7267	+0.0000	+0.00%
Official checkpoint reproduction	6.07	5.74	4.90	5.13	5.40	4.90	3.58	3.51	3.29	4.7244	-0.0022	-0.05%
This run step_3000	5.59	5.10	4.42	4.36	4.64	4.28	2.97	2.85	2.68	4.0989	-0.6278	-13.28%
This run step_6000	5.78	5.36	4.50	4.69	4.95	4.58	3.26	3.14	2.93	4.3544	-0.3723	-7.88%
This run step_9000	5.92	5.48	4.66	4.89	5.17	4.70	3.40	3.30	3.07	4.5100	-0.2167	-4.58%
This run step_12000	6.04	5.60	4.88	4.98	5.25	4.75	3.48	3.38	3.18	4.6156	-0.1111	-2.35%
This run step_15000	6.07	5.59	4.74	5.08	5.38	4.83	3.62	3.47	3.21	4.6656	-0.0611	-1.29%
This run step_18000（published）	6.12	5.67	4.99	5.12	5.40	4.91	3.64	3.50	3.29	4.7378	+0.0111	+0.24%
This run step_21000	6.05	5.72	4.94	5.17	5.44	4.89	3.60	3.55	3.32	4.7422	+0.0156	+0.33%
This run step_24000	6.09	5.73	4.83	5.16	5.47	4.89	3.65	3.56	3.29	4.7411	+0.0144	+0.31%
This run step_27000	6.05	5.74	4.90	5.10	5.45	4.94	3.64	3.56	3.32	4.7444	+0.0177	+0.38%
This run step_27110	6.06	5.75	4.96	5.14	5.47	4.95	3.67	3.57	3.33	4.7667	+0.0400	+0.85%

The published step 18,000 checkpoint slightly exceeds the paper's nine-task mean by 0.0111 accepted tokens（+0.24%）and matches the paper's Arena-Hard accepted length of 3.29. Later checkpoints mostly plateau, while the final step 27,110 checkpoint reaches the highest observed mean of 4.7667.

Published checkpoint: full evaluation results

The step 18,000 checkpoint was evaluated on four GPUs with the fixed protocol above. AR@n is the acceptance rate at proposal position n.

Dataset	Samples	Proposals	Accept length	Verify rate	AR@0	AR@1	AR@2	AR@3	AR@4	AR@5	AR@6
GSM8K	500	6.95+1	6.12	0.7696	0.9321	0.8638	0.7964	0.7317	0.6711	0.6134	0.5599
MATH-500	500	6.98+1	5.67	0.7095	0.9075	0.8144	0.7286	0.6528	0.5856	0.5243	0.4662
AIME 2025	30	7.00+1	4.99	0.6240	0.8761	0.7508	0.6401	0.5465	0.4624	0.3892	0.3270
HumanEval	164	6.96+1	5.40	0.6777	0.8981	0.7960	0.6987	0.6164	0.5406	0.4697	0.4073
MBPP	256	6.94+1	5.12	0.6441	0.8844	0.7726	0.6672	0.5727	0.4913	0.4162	0.3541
LiveCodeBench	500	6.95+1	4.91	0.6177	0.8692	0.7497	0.6365	0.5388	0.4557	0.3806	0.3176
MT-Bench	80	6.97+1	3.64	0.4566	0.7687	0.5759	0.4248	0.3169	0.2364	0.1854	0.1462
Alpaca	500	6.97+1	3.50	0.4391	0.7655	0.5668	0.4100	0.2951	0.2128	0.1543	0.1100
Arena-Hard-v2	500	6.99+1	3.29	0.4118	0.6678	0.4828	0.3588	0.2739	0.2128	0.1672	0.1315

Confidence-head calibration

Confidence metrics were collected with early stopping disabled （confidence_threshold=0.0）. ECE and Brier score are lower-is-better; AUC is higher-is-better. Pred mean and target mean are mean predicted confidence and empirical acceptance respectively.

Step 18,000 by dataset

Dataset	Samples	Proposals	ECE	AUC	Brier	Pred mean	Target mean
GSM8K	500	24,560	0.0659	0.8850	0.1179	0.8048	0.7390
MATH-500	500	71,813	0.0325	0.8888	0.1175	0.7004	0.6687
AIME 2025	30	11,065	0.0175	0.8677	0.1292	0.5757	0.5703
HumanEval	164	13,325	0.0761	0.8772	0.1323	0.7083	0.6330
MBPP	256	16,119	0.0584	0.8624	0.1383	0.6523	0.5950
LiveCodeBench	500	29,063	0.0682	0.8677	0.1395	0.6322	0.5649
MT-Bench	80	10,378	0.0214	0.8710	0.1200	0.3970	0.3797
Alpaca	500	52,345	0.0271	0.8484	0.1254	0.3868	0.3599
Arena-Hard-v2	500	155,497	0.0203	0.9103	0.1011	0.3465	0.3280

Usage with DeepSpec

Install the current DeepSpec implementation and use this repository as the draft checkpoint. The tokenizer comes from the target model and is therefore not duplicated here.

git clone https://github.com/deepseek-ai/DeepSpec.git
cd DeepSpec
# Install the dependencies described by DeepSpec, then run:
CUDA_VISIBLE_DEVICES=0,1,2,3 python eval.py \
  --target_name_or_path Qwen/Qwen3-4B \
  --draft_name_or_path alice1001/dspark_qwen3_4b_block7 \
  --max-new-tokens 2048 \
  --temperature 1.0 \
  --confidence-threshold 0.0 \
  --seed 980406

Limitations

• This is a draft model for Qwen3-4B speculative decoding, not a replacement for the target language model.
• It was trained on non-thinking Qwen3-4B generations; behavior may degrade for thinking-mode or strongly shifted domains.
• The repository contains step 18,000, not the final step 27,110 checkpoint. Later-step metrics are included only to document the full training run.
• Reported values depend on the exact target model, sampling settings, DeepSpec revision, hardware, and benchmark prompts.
• Runtime acceptance metrics do not establish benchmark task accuracy.

License and attribution

The base Qwen3-4B model is released under Apache-2.0. DeepSpec code is released under MIT. Users should review and comply with both upstream projects and the licenses of the training and evaluation datasets.