spikes/002a-mini-gpu-smoke/README.md

Spike 002a-mini — Real GPU Smoke

Closes: cross-model review item #4 (zero GPU evidence anywhere) + ADR-001's choice of local 5090 over Modal.

Goal

Take Spike 006's CPU smoke and run it on real GPU hardware to confirm:

• bf16 numerics work end-to-end through the 3-channel loss
• VRAM usage is well-bounded on a 0.5B model
• Step time is stable on the local 5090 (no thermal throttling, no swap)
• The framework's design choices (mixed-precision compatibility, GPU dtype casts, etc) hold on real hardware, not just CPU.

Setup

• Hardware: local NVIDIA RTX 5090 (Blackwell sm_120, 32 GB VRAM)
• Software: torch 2.12.0+cu130, transformers 4.57.6, fp32 not used (we go straight to bf16 — the modern default for 0.5B models)
• Model: Qwen/Qwen2.5-0.5B-Instruct (the same model as Spike 006 CPU smoke, for direct CPU↔GPU comparison)

Run

cd spikes/002a-mini-gpu-smoke
python run_gpu_smoke.py

Default: 50 steps × composer_total_loss × Qwen2.5-0.5B-Instruct on device='cuda', dtype=bf16. Captures per-step memory + step-time + finite-grads check + monotonic loss-decrease check + peak-VRAM bound check.

What this verifies (and what it doesn't)

VERIFIES:

• Real model loads on real GPU
• 3-channel loss runs end-to-end through bf16
• Peak VRAM is well under headroom (5.31 GB on 0.5B model with bf16)
• Step time is stable (no warmup churn after step 0)
• Loss decreases meaningfully (>50% reduction over 50 steps)

DOES NOT VERIFY:

• That the model is being trained correctly (this is a verification harness, not a real GRPO run — see Spike 006-strict for the SDPO channel exercise + the production path via ComposerReplicationTrainer)
• That training produces Composer-2.5-quality results (post-replication GPU phase, requires real teacher rollouts)
• Multi-GPU or multi-replica DiLoCo (Spike 008 single-process limitation applies; multi-process DiLoCo is post-replication work)

Cost

• $0 (local 5090, no Modal spend per ADR-001)
• 35 s wall-clock total
• 5.31 GB peak VRAM

Files

• run_gpu_smoke.py — runner
• verdict.md — pass/fail summary with metrics
• results/gpu_loss_curve.csv — per-step metrics
• results/gpu_verdict.json — programmatic verdict