spikes/002a-mini-gpu-smoke/README.md

# Spike 002a-mini — Real GPU Smoke

**Closes**: cross-model review item #4 (zero GPU evidence anywhere) +
ADR-001's choice of local 5090 over Modal.

## Goal

Take Spike 006's CPU smoke and run it on real GPU hardware to confirm:
- bf16 numerics work end-to-end through the 3-channel loss
- VRAM usage is well-bounded on a 0.5B model
- Step time is stable on the local 5090 (no thermal throttling, no swap)
- The framework's design choices (mixed-precision compatibility, GPU
  dtype casts, etc) hold on real hardware, not just CPU.

## Setup

- **Hardware**: local NVIDIA RTX 5090 (Blackwell sm_120, 32 GB VRAM)
- **Software**: torch 2.12.0+cu130, transformers 4.57.6, fp32 not used (we
  go straight to bf16 — the modern default for 0.5B models)
- **Model**: `Qwen/Qwen2.5-0.5B-Instruct` (the same model as Spike 006
  CPU smoke, for direct CPU↔GPU comparison)

## Run

```bash
cd spikes/002a-mini-gpu-smoke
python run_gpu_smoke.py
```

Default: 50 steps × `composer_total_loss` × Qwen2.5-0.5B-Instruct on
device='cuda', dtype=bf16. Captures per-step memory + step-time + finite-grads
check + monotonic loss-decrease check + peak-VRAM bound check.

## What this verifies (and what it doesn't)

VERIFIES:
- Real model loads on real GPU
- 3-channel loss runs end-to-end through bf16
- Peak VRAM is well under headroom (5.31 GB on 0.5B model with bf16)
- Step time is stable (no warmup churn after step 0)
- Loss decreases meaningfully (>50% reduction over 50 steps)

DOES NOT VERIFY:
- That the model is being trained correctly (this is a verification
  harness, not a real GRPO run — see Spike 006-strict for the SDPO
  channel exercise + the production path via `ComposerReplicationTrainer`)
- That training produces Composer-2.5-quality results (post-replication
  GPU phase, requires real teacher rollouts)
- Multi-GPU or multi-replica DiLoCo (Spike 008 single-process limitation
  applies; multi-process DiLoCo is post-replication work)

## Cost

- $0 (local 5090, no Modal spend per ADR-001)
- 35 s wall-clock total
- 5.31 GB peak VRAM

## Files

- `run_gpu_smoke.py` — runner
- `verdict.md` — pass/fail summary with metrics
- `results/gpu_loss_curve.csv` — per-step metrics
- `results/gpu_verdict.json` — programmatic verdict