Daimon / specs /runpod_multigpu_options.md
HumboldtJoker's picture
Add spec: runpod_multigpu_options.md
ce5d8d4 verified
|
Raw
History Blame Contribute Delete
2.41 kB

RunPod Multi-GPU Options for Qwen3.6-35B-A3B Full SFT

Requirement: 160GB+ VRAM, 500GB+ system RAM, 560GB+ total memory.

Best Single-Pod Options (ranked)

Config VRAM Rate (Community) Rate (Secure) Notes
4x A100 80GB SXM 320GB ~$5.56/hr ~$6.80/hr Best fit. VRAM headroom reduces CPU offload. NVLink 600GB/s. Often "Unavailable" -- rotates in/out.
2x H100 80GB SXM 160GB ~$5.38/hr ~$6.58/hr Minimum viable VRAM. NVLink 900GB/s. Better availability in US regions.
2x A100 80GB SXM 160GB ~$2.78/hr ~$3.40/hr Cheapest viable. Tight on VRAM -- heavy CPU offload via DeepSpeed ZeRO-3 required.
4x H100 80GB SXM 320GB ~$10.76/hr ~$13.16/hr Overkill but fast. Rarely unavailable in Secure US.

System RAM is configurable per pod. Request 502GB+ when creating the pod -- RunPod allows custom RAM allocation up to host limits. SXM variants are mandatory for NVLink (PCIe has no inter-GPU fabric).

Instant Clusters (Multi-Node)

RunPod Instant Clusters link 2-8 nodes (up to 64 GPUs) with 800Gbps-3.2Tbps inter-node bandwidth. H100 SXM focused. Pre-configured with distributed PyTorch env vars and Slurm support. Setup takes minutes.

  • 2-node cluster (2x 8xH100 SXM): ~$43/hr but guaranteed availability and 128x80GB = 10.24TB aggregate memory
  • Use case: If single-pod 4xA100 stays unavailable, a 2-node cluster with fewer GPUs per node may be obtainable faster

Docs: https://docs.runpod.io/instant-clusters

Grabbing Scarce Pods

RunPod has no built-in waitlist or availability alerts. Strategies:

  1. Poll the API -- runpodctl or REST API to check GPU type availability programmatically; script a notification
  2. Try off-peak -- US availability best outside 9am-5pm PT weekdays
  3. Secure Cloud > Community Cloud -- US Secure regions stay available "almost always" for A100/H100; Community is cheaper but host-dependent
  4. Region hop -- 30+ regions; EU-CENTRAL H100 SXM scarce during business hours, US-TX/US-OR better

Recommendation

Start with 2x A100 80GB SXM Secure Cloud (~$2.78/hr) + DeepSpeed ZeRO-3 offload. If VRAM is too tight, upgrade to 4x A100 SXM when it appears. Write a 20-line polling script against RunPod API to alert on 4xA100 availability. Instant Clusters are the fallback if single-pod multi-GPU stays scarce.