docs/adrs/ADR-001-gpu-venue.md

# ADR-001 — GPU venue for Spike 002a-mini smoke

**Status**: Accepted
**Date**: 2026-05-26
**Wave**: Phase 4 (deep work loop)

## Context

Spike 002a-mini is the optional Phase-10 gate in the deep work loop: take the
real-HF-model loss-composition smoke (Spike 006) and run it on GPU to confirm
bf16 numerics, capture memory + step-time, and rule out CPU-only blind spots
before publishing the framework.

The user has:
- a 5090 (32 GB VRAM, Blackwell) on the local box (this WSL host)
- a configured Modal account (~/.modal.toml present, modal CLI installed)

The workload:
- `Qwen/Qwen2.5-0.5B-Instruct` (~1 GB bf16 weights)
- ~50 forward+backward steps through the 3-channel loss
- single GPU, no distributed training, no FSDP

## Options considered

### Option A — Local 5090

- Free, no rate limit, no cold start.
- Iteration loop: code change → run → fix → run is **~25-40 s wall-clock per cycle**.
- 32 GB VRAM ≫ 24 GB needed for this workload.
- WSL CUDA path is the same one we use for eidolon training already; toolchain proven.
- Reuses local HF cache (~/.cache/huggingface), no re-download per run.

### Option B — Modal L4 ($0.000222/sec ≈ $0.799/hr)

- $0.08-0.13 per smoke run (3-7 min wall-clock incl cold start).
- Iteration loop: code change → modal-run dispatch → image build (cached) → cold
  start → run → modal volume get → fix is **~3-5 min per cycle even on a cache hit**.
- Persistent volume saves model re-download across runs.
- Decoupled from local environment state.
- Extensively documented gotchas in `mlops/modal-llm-training` skill (M1-M9).

### Option C — Modal A100-40GB

- ~3× cost of L4 for 0.5B workload that doesn't need the capacity. Ruled out.

## Decision

**Option A — local 5090.** The 5090 dominates Modal L4 on every dimension that
matters for a 0.5B sub-1B-param verification smoke:

| Dimension | 5090 (local) | Modal L4 |
|---|---|---|
| Iteration cycle | 25-40 s | 3-5 min (10× slower) |
| $ / smoke run | $0 | $0.10 |
| VRAM headroom | 32 GB > 24 GB needed | 24 GB ≈ 24 GB needed |
| State decoupling | Same machine as dev | Decoupled (advantage Modal) |
| Toolchain risk | Already proven | New for this workload |

The "decoupled state" advantage of Modal is real but doesn't outweigh the 10×
iteration penalty for what is fundamentally a verification step. We're not
running production training; we're checking that a GPU run agrees with the
CPU run we just did.

## Consequences

### Accepted

- Spike 002a-mini becomes a **local 5090** smoke, not a Modal job.
- The `mlops/modal-llm-training` skill's L4 pattern (modal_app.py skeleton in
  `docs/research/MODAL_RECONNAISSANCE.md`) is **stashed for future use** — it's
  the right pattern when we DO need cloud GPU.
- `docs/research/MODAL_RECONNAISSANCE.md` stays in the repo as the design
  document for the Modal path; the file documents *why* we didn't use Modal
  for this smoke and *when* Modal becomes correct.

### Modal becomes the right choice when

1. **Parallel parameter sweeps** — N independent runs across α, β, lr, etc.
   that need to fan out faster than wall-clock-sequential on a single 5090.
2. **Scaling to ≥7B base models** — 5090's 32 GB starts to bind on 7B + LoRA
   + activation memory at seq 4096+. A100-40 or H100 becomes necessary.
3. **Multi-node training** — DiLoCo-style outer-loop across 2+ physical
   nodes for the eventual full RL run.
4. **CI / reproducibility** — a future contributor wants to repro our results
   without owning a 5090.

These are all **post-replication** workloads. The deep work loop's gap-closer
phase (W7-W10) doesn't need any of them.

### Trade-offs explicitly accepted

- We carry one local-environment dependency (WSL CUDA + the 5090 driver) that
  Modal would have absorbed. Mitigated by: the same dependency is already
  exercised by eidolon training, so the marginal risk is zero.
- We don't get an audit-friendly "Modal app run with persistent receipts"
  artifact. Mitigated by: capturing `nvidia-smi` snapshots + step-time CSV
  into `spikes/006-real-hf-model-smoke/results/` as our local audit trail.

## Source

`docs/research/MODAL_RECONNAISSANCE.md` (subagent recon, primary-sourced from
modal.com/pricing and modal.com/docs, 2026-05-26).