yushengsu/dense-1-8

dense-1/8: a small Qwen3-8B DSpark dense drafter (2 layers, 1/4 attention width, 1/8 MLP)

Per-epoch checkpoints of one SpecForge run: the Qwen3-8B dense DSpark drafter (causal draft block, block 16, no confidence head) shrunk in structure only: 2 layers instead of 5, attention GQA 8/2 heads x 128 (1/4 of the dense arm's width), MLP intermediate_size 1536 (1/8 of 12288); residual hidden stays 4096. About 0.22B parameters. Trained for 12 epochs on the Open Perfect Blend corpus regenerated by Qwen3-8B (yaro1214/open_data), rendered with the qwen3-instruct template and no system prompt.

• Code, config and the per-epoch results table: https://github.com/yushengsu-thu/SpecForge/blob/dense-1/8/exp_read.me
• Training curve (one W&B line for all 12 epochs): https://wandb.ai/eigent_radixark_training/specforge-qwen3-8b-dspark-dense

Layout: epoch-01/ ... epoch-12/, each a specforge exported servable directory (step 10764 x epoch of the run; global batch 132 on 6 DDP ranks). Folders appear as epochs finish.