ADR-001 — GPU venue for Spike 002a-mini smoke

Status: Accepted Date: 2026-05-26 Wave: Phase 4 (deep work loop)

Context

Spike 002a-mini is the optional Phase-10 gate in the deep work loop: take the real-HF-model loss-composition smoke (Spike 006) and run it on GPU to confirm bf16 numerics, capture memory + step-time, and rule out CPU-only blind spots before publishing the framework.

The user has:

a 5090 (32 GB VRAM, Blackwell) on the local box (this WSL host)
a configured Modal account (~/.modal.toml present, modal CLI installed)

The workload:

Qwen/Qwen2.5-0.5B-Instruct (~1 GB bf16 weights)
~50 forward+backward steps through the 3-channel loss
single GPU, no distributed training, no FSDP

Options considered

Option A — Local 5090

Free, no rate limit, no cold start.
Iteration loop: code change → run → fix → run is ~25-40 s wall-clock per cycle.
32 GB VRAM ≫ 24 GB needed for this workload.
WSL CUDA path is the same one we use for eidolon training already; toolchain proven.
Reuses local HF cache (~/.cache/huggingface), no re-download per run.

Option B — Modal L4 ($0.000222/sec ≈ $0.799/hr)

$0.08-0.13 per smoke run (3-7 min wall-clock incl cold start).
Iteration loop: code change → modal-run dispatch → image build (cached) → cold start → run → modal volume get → fix is ~3-5 min per cycle even on a cache hit.
Persistent volume saves model re-download across runs.
Decoupled from local environment state.
Extensively documented gotchas in mlops/modal-llm-training skill (M1-M9).

Option C — Modal A100-40GB

~3× cost of L4 for 0.5B workload that doesn't need the capacity. Ruled out.

Decision

Option A — local 5090. The 5090 dominates Modal L4 on every dimension that matters for a 0.5B sub-1B-param verification smoke:

Dimension	5090 (local)	Modal L4
Iteration cycle	25-40 s	3-5 min (10× slower)
$ / smoke run	$0	$0.10
VRAM headroom	32 GB > 24 GB needed	24 GB ≈ 24 GB needed
State decoupling	Same machine as dev	Decoupled (advantage Modal)
Toolchain risk	Already proven	New for this workload

The "decoupled state" advantage of Modal is real but doesn't outweigh the 10× iteration penalty for what is fundamentally a verification step. We're not running production training; we're checking that a GPU run agrees with the CPU run we just did.

Consequences

Accepted

Spike 002a-mini becomes a local 5090 smoke, not a Modal job.
The mlops/modal-llm-training skill's L4 pattern (modal_app.py skeleton in docs/research/MODAL_RECONNAISSANCE.md) is stashed for future use — it's the right pattern when we DO need cloud GPU.
docs/research/MODAL_RECONNAISSANCE.md stays in the repo as the design document for the Modal path; the file documents why we didn't use Modal for this smoke and when Modal becomes correct.

Modal becomes the right choice when

Parallel parameter sweeps — N independent runs across α, β, lr, etc. that need to fan out faster than wall-clock-sequential on a single 5090.
Scaling to ≥7B base models — 5090's 32 GB starts to bind on 7B + LoRA
- activation memory at seq 4096+. A100-40 or H100 becomes necessary.
Multi-node training — DiLoCo-style outer-loop across 2+ physical nodes for the eventual full RL run.
CI / reproducibility — a future contributor wants to repro our results without owning a 5090.

These are all post-replication workloads. The deep work loop's gap-closer phase (W7-W10) doesn't need any of them.

Trade-offs explicitly accepted

We carry one local-environment dependency (WSL CUDA + the 5090 driver) that Modal would have absorbed. Mitigated by: the same dependency is already exercised by eidolon training, so the marginal risk is zero.
We don't get an audit-friendly "Modal app run with persistent receipts" artifact. Mitigated by: capturing nvidia-smi snapshots + step-time CSV into spikes/006-real-hf-model-smoke/results/ as our local audit trail.

Source

docs/research/MODAL_RECONNAISSANCE.md (subagent recon, primary-sourced from modal.com/pricing and modal.com/docs, 2026-05-26).