spikes/002a-mini-gpu-smoke/verdict.md

Spike 002a-mini-gpu-smoke — VERDICT

Status: ✅ PASSED on local 5090 Date: 2026-05-26 Wave: 12 (closing the "zero GPU evidence" gap from cross-model review item #4)

Headline

composer_replication 3-channel loss composition runs cleanly on real GPU hardware. Qwen2.5-0.5B-Instruct on RTX 5090 sm_120 in bf16, 50 backward steps, loss 0.7354 → 0.00034 (99.95% reduction), all gradients finite throughout. Peak VRAM 5.31 GB (well under the ADR-001 8GB target). Median step time 480ms.

Closes

• The cross-model review's item #4: "Run Spike 002a-mini on the local 5090. ADR-001 made the choice; the spike was not run. Until then, the framework has zero GPU evidence of any kind." Done.
• ADR-001's underlying claim that local 5090 is the right venue for this workload class. Verified: 50-step run completes in ~30 s wall-clock on the local 5090, vs an estimated 3-5 min cold-start cycle on Modal L4.
• The "but the framework only runs on CPU" objection in V8.

Acceptance criteria

Criterion	Target	Result
Model loads via AutoModelForCausalLM on cuda	bf16, no errors	✅ 7.3 s
50 steps run end-to-end	No nan/inf	✅
Loss decreases meaningfully	final < 50% × initial	✅ final = 0.046% × initial
Peak VRAM < 8 GB on 0.5B model	headroom check	✅ 5.31 GB
Step time stable	no thermal throttling, no swap	✅ median 480ms, no outliers
All gradients finite throughout	per-step finite check	✅
sm_120 Blackwell architecture supported	not pre-Hopper-only	✅ verified arch in torch.cuda.get_arch_list()

Per-channel behavior on GPU

Same as CPU (Spike 006): LM-CE channel dominates, DPO channel contributes small nonzero gradient throughout, SDPO channel zero (shape-mismatch fallback — to exercise the SDPO channel on GPU, run with align_sdpo_shapes batch builder per Spike 006-strict's test_sdpo_channel_actually_fires).

Memory profile

step	total	peak_mem_gb	step_time_ms
0 (post-warmup)	0.7354	5.31	~500
10	0.0067	5.31	~480
25	0.0007	5.31	~480
49	0.0003	5.31	~480

Memory stays flat at 5.31 GB after warmup — no leak, no expanding activation buffers. (The 0.5B model in bf16 + Adam states + activations + DPO logit gradients all fit comfortably.)

What this does NOT close

• Multi-replica / multi-process DiLoCo (V2 partial gap). This spike is single-GPU. Real DiLoCo training across replicas is GPU-multi-process and not yet attempted.
• Composer-2.5-quality empirical results (V8 partial gap). This spike verifies the framework runs on GPU; it does NOT verify the method improves model quality vs plain GRPO. That requires the full pipeline (real teacher rollouts + real GRPO rewards + a benchmark like SWE-bench-lite) and is the post-replication GPU phase ($30-100+).

Files

• run_gpu_smoke.py — 50-step GPU smoke runner with VRAM + step-time capture
• results/gpu_loss_curve.csv — per-step metrics
• results/gpu_verdict.json — programmatic verdict for CI/audit
• results/run.log — actual successful run output

Cost / time

• $0 (local 5090, no Modal spend)
• 35 s wall-clock total (7 s model load + 25 s training)
• ~5 GB VRAM