Moonlight556/qwen3-8b-dflash-mathregen-sweep

Qwen3-8B DFlash math-regen LR × loss-type sweep — final checkpoints

Trained from mem-research/specforge branch dev/math-regen-sweep. Target model: Qwen/Qwen3-8B (frozen). Draft head: causal parallel drafter (--causal-head), trained with Jacobi-Forcing-style objective on the math-regen dataset (memset0/nemotron-v2-math-qwen3-8b-nothink).

Sweep grid: 3 LRs × 2 loss types = 6 cells, 2 epochs each, batch 2 × accum 2 × 4 GPUs (effective batch 16), seqlen 3072, num-anchors 512, warmup-ratio 0.04 cosine.

Subdir	LR	Loss type	WandB run	Final accept_len
cell_1_lr3e-4_gt/	3e-4	teacher-force	rxn4txue	9.805
cell_2_lr1e-4_gt/	1e-4	teacher-force	knirzoye	(see WandB)
cell_3_lr5e-5_gt/	5e-5	teacher-force	owb9jbvo	(see WandB)
cell_4_lr3e-4_distill/	3e-4	symmetric soft KD (T=1)	t96ysxm6	(see WandB)
cell_5_lr1e-4_distill/	1e-4	symmetric soft KD (T=1)	olm6rm48	(see WandB)
cell_6_lr5e-5_distill/	5e-5	symmetric soft KD (T=1)	pv7pg0cs	(still training)

Each subdir contains:

• config.json — draft head config
• dflash.py — draft head model code
• model.safetensors — draft head weights (2.1 GB)
• training_state.pt — optimizer/scheduler/RNG state (4.6 GB; needed only to resume training, not for eval)

WandB project: https://wandb.ai/lanxiang_llm/specforge-qwen3-8b-dflash.