Add spec: runpod_multigpu_options.md

specs/runpod_multigpu_options.md

@@ -0,0 +1,36 @@
+# RunPod Multi-GPU Options for Qwen3.6-35B-A3B Full SFT
+
+**Requirement:** 160GB+ VRAM, 500GB+ system RAM, 560GB+ total memory.
+
+## Best Single-Pod Options (ranked)
+
+| Config | VRAM | Rate (Community) | Rate (Secure) | Notes |
+|--------|------|-------------------|----------------|-------|
+| 4x A100 80GB SXM | 320GB | ~$5.56/hr | ~$6.80/hr | Best fit. VRAM headroom reduces CPU offload. NVLink 600GB/s. Often "Unavailable" -- rotates in/out. |
+| 2x H100 80GB SXM | 160GB | ~$5.38/hr | ~$6.58/hr | Minimum viable VRAM. NVLink 900GB/s. Better availability in US regions. |
+| 2x A100 80GB SXM | 160GB | ~$2.78/hr | ~$3.40/hr | Cheapest viable. Tight on VRAM -- heavy CPU offload via DeepSpeed ZeRO-3 required. |
+| 4x H100 80GB SXM | 320GB | ~$10.76/hr | ~$13.16/hr | Overkill but fast. Rarely unavailable in Secure US. |
+
+System RAM is configurable per pod. Request 502GB+ when creating the pod -- RunPod allows custom RAM allocation up to host limits. SXM variants are mandatory for NVLink (PCIe has no inter-GPU fabric).
+
+## Instant Clusters (Multi-Node)
+
+RunPod Instant Clusters link 2-8 nodes (up to 64 GPUs) with 800Gbps-3.2Tbps inter-node bandwidth. H100 SXM focused. Pre-configured with distributed PyTorch env vars and Slurm support. Setup takes minutes.
+
+- **2-node cluster (2x 8xH100 SXM):** ~$43/hr but guaranteed availability and 128x80GB = 10.24TB aggregate memory
+- **Use case:** If single-pod 4xA100 stays unavailable, a 2-node cluster with fewer GPUs per node may be obtainable faster
+
+Docs: https://docs.runpod.io/instant-clusters
+
+## Grabbing Scarce Pods
+
+RunPod has **no built-in waitlist or availability alerts**. Strategies:
+
+1. **Poll the API** -- `runpodctl` or REST API to check GPU type availability programmatically; script a notification
+2. **Try off-peak** -- US availability best outside 9am-5pm PT weekdays
+3. **Secure Cloud > Community Cloud** -- US Secure regions stay available "almost always" for A100/H100; Community is cheaper but host-dependent
+4. **Region hop** -- 30+ regions; EU-CENTRAL H100 SXM scarce during business hours, US-TX/US-OR better
+
+## Recommendation
+
+Start with **2x A100 80GB SXM Secure Cloud** (~$2.78/hr) + DeepSpeed ZeRO-3 offload. If VRAM is too tight, upgrade to **4x A100 SXM** when it appears. Write a 20-line polling script against RunPod API to alert on 4xA100 availability. Instant Clusters are the fallback if single-pod multi-GPU stays scarce.