Add spec: runpod_oom_debug.md

specs/runpod_oom_debug.md

@@ -0,0 +1,349 @@
+# RunPod H200 SXM SIGKILL Debug: Qwen3.6-35B-A3B Full SFT
+
+**Date:** 2026-06-30
+**Pod:** vocal_coral_quokka (jifd0w19x1c61j)
+**Status:** Root cause identified. Full SFT is not viable on this pod configuration.
+
+---
+
+## Root Cause: Container Memory Limit, Not Host RAM
+
+### The Lie `free -h` Tells You
+
+RunPod containers see **host RAM** (2TB) in `/proc/meminfo` and `free -h`, but the
+actual memory available to the container is enforced by **cgroup limits** that are
+much lower. For a 1x H200 SXM pod, RunPod allocates:
+
+| Resource        | Spec per 1x H200 SXM |
+|-----------------|----------------------|
+| VRAM            | 141 GB HBM3e         |
+| **System RAM**  | **276 GB**           |
+| vCPUs           | 24                   |
+
+**The container is capped at ~276 GB system RAM.** When a process exceeds this,
+the Linux OOM killer sends SIGKILL (-9). There is no warning, no OOM error message,
+just instant death. This matches the observed behavior exactly.
+
+### How to Verify Inside the Pod
+
+```bash
+# cgroup v1 (most RunPod pods)
+cat /sys/fs/cgroup/memory/memory.limit_in_bytes
+
+# cgroup v2
+cat /sys/fs/cgroup/memory.max
+
+# What the process THINKS is available (WRONG - shows host RAM)
+free -h
+
+# What's actually enforced (check OOM events)
+dmesg | grep -i "oom\|killed"
+```
+
+---
+
+## Memory Math: Why Full SFT Cannot Fit
+
+### Qwen3.6-35B-A3B Architecture
+
+- **Total parameters:** 35B (35 billion)
+- **Active per token:** 3B (8 routed + 1 shared expert of 256 total)
+- **Layers:** 40
+- **Experts:** 256 per MoE layer, 9 active
+- **Hidden dim:** 2,048
+- **Architecture:** Sparse MoE — inference is cheap, but **training touches ALL 35B params**
+
+### The Critical MoE Training Problem
+
+During inference, only 3B parameters are active per token. But during **training**:
+- **Forward pass:** routes through selected experts (cheap)
+- **Backward pass:** computes gradients for ALL activated expert parameters
+- **Optimizer step:** maintains states for **ALL 35B parameters** regardless of activation
+
+This means training memory scales with TOTAL params (35B), not active params (3B).
+
+### Memory Breakdown for Full SFT (Single GPU)
+
+| Component                  | Calculation           | Size     |
+|----------------------------|-----------------------|----------|
+| Model weights (bf16)       | 35B x 2 bytes         | **70 GB**  |
+| fp32 master weights        | 35B x 4 bytes         | **140 GB** |
+| AdamW momentum (fp32)      | 35B x 4 bytes         | **140 GB** |
+| AdamW variance (fp32)      | 35B x 4 bytes         | **140 GB** |
+| Gradients (bf16)           | 35B x 2 bytes         | **70 GB**  |
+| **Total (steady state)**   |                       | **560 GB** |
+| Activation memory          | batch/seq dependent   | 10-50 GB+  |
+| CUDA/PyTorch overhead      | fragmentation, buffers| 5-15 GB  |
+| **Peak during init**       | temporary copies      | **600-700 GB** |
+
+### What Each ZeRO Stage Actually Requires
+
+**ZeRO-2, no offload (attempt 1):**
+- GPU needs: 70 (weights) + 140 (master) + 280 (optimizer) + 70 (grads) = **560 GB**
+- Available GPU: 141 GB
+- Result: `torch.OutOfMemoryError` trying to allocate 129 GB. Correct behavior.
+
+**ZeRO-2, CPU offload (attempt 2):**
+- GPU: ~70 GB model weights + activations + grad buffers ≈ 100-130 GB
+- CPU: fp32 master weights (140 GB) + optimizer states (280 GB) = **420 GB**
+- Available CPU (container): **276 GB**
+- Deficit: **~144 GB over limit**
+- Result: SIGKILL during init. 100% CPU for 2 min = optimizer state allocation eating RAM until OOM killer fires.
+
+**ZeRO-3, CPU offload (attempt 3):**
+- GPU: only active parameter slice + activations ≈ 20-40 GB
+- CPU: partitioned states BUT with 1 GPU, nothing to partition across
+- CPU needs: same ~420-560 GB for optimizer + master weights
+- **Additionally:** ZeRO-3 does NOT support MoE models. DeepSpeed raises `AssertionError: MoE not supported with Stage 3` in current versions.
+- Even if it initialized, peak init memory creates temporary copies that spike higher
+- Result: SIGKILL after ~4 min of init. Same container RAM ceiling.
+
+### The ZeRO-3 + MoE Incompatibility
+
+This is a confirmed, documented limitation:
+- **GitHub Issue #2870** (filed Feb 2023, still open): "ZeRO stage 3 support for mixture-of-experts (MoE) layer" — explicitly unsupported
+- **GitHub Issue #7156** (open): Even with ZeRO-2, expert optimizer states are NOT partitioned — only non-expert params get ZeRO treatment
+- The MoE expert parameters maintain full unpartitioned optimizer states, making them the dominant memory consumer
+
+### The ms-swift Confirmation
+
+GitHub Issue **modelscope/ms-swift#6473** documents the identical failure:
+- **Same model family:** Qwen3-30B-A3B (MoE)  
+- **Same setup:** DeepSpeed ZeRO-2 with CPU offload
+- **Same result:** System RAM climbed to 1.7 TiB, OOM killed
+- **Hardware:** 8x H200 GPUs, 1.5TB RAM, 96 CPU cores
+- **Key insight:** Even with 8 GPUs and 1.5TB RAM, full SFT of this MoE model exhausted all system memory
+
+If 8x H200s with 1.5TB RAM cannot do this, 1x H200 with 276 GB has zero chance.
+
+---
+
+## Viable Alternatives (Ranked by Practicality)
+
+### Option 1: LoRA Fine-Tuning with Unsloth (RECOMMENDED)
+
+**Why it works:** LoRA only trains adapter weights (~2-3% of params), so optimizer states
+are proportional to LoRA rank, not total model params.
+
+| Metric                     | Value                  |
+|----------------------------|------------------------|
+| Trainable params           | ~931M of 36B (2.58%)   |
+| VRAM required (bf16 LoRA)  | ~63-74 GB              |
+| System RAM required        | ~40-80 GB              |
+| Fits on 1x H200 SXM?      | **Yes** (141 GB VRAM)  |
+| Framework                  | Unsloth or ms-swift    |
+
+**Unsloth Configuration:**
+```python
+from unsloth import FastLanguageModel
+
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name="Qwen/Qwen3.6-35B-A3B",
+    max_seq_length=2048,        # Safe ceiling for 1 GPU
+    dtype=torch.bfloat16,
+    load_in_4bit=False,         # QLoRA NOT recommended for MoE (BnB limitation)
+)
+
+model = FastLanguageModel.get_peft_model(
+    model,
+    r=16,                       # LoRA rank
+    lora_alpha=16,              # alpha == r, not r*2
+    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
+                    "gate_proj", "up_proj", "down_proj"],
+    lora_dropout=0,
+    use_gradient_checkpointing="unsloth",
+)
+```
+
+**Critical Gotchas:**
+1. Set `UNSLOTH_COMPILE_DISABLE=1` before imports (bf16/fp32 dtype mismatch in compiled MoE kernels)
+2. Set `dataloader_num_workers=0` and `dataset_num_proc=1` (multiprocessing deadlock with large model)
+3. Do NOT use QLoRA/4-bit — BitsandBytes does not support MoE architectures
+4. Router layer is NOT fine-tuned by default (and shouldn't be for SFT)
+5. Max safe sequence length on single H200: ~2048 tokens (activation memory explodes at 4096)
+6. vLLM cannot load LoRA adapters directly — merge weights before serving
+
+**Unsloth Split LoRA** provides additional MoE-specific optimization: reorders matmul
+to compute `(X @ loraA) @ loraB` instead of materializing `loraA @ loraB`, saving ~35%
+VRAM and providing ~2x speedup.
+
+### Option 2: Multi-GPU with Expert Parallelism (Megatron-SWIFT)
+
+**Why it works:** Distributes expert params AND their optimizer states across GPUs.
+Each GPU only holds a subset of experts.
+
+| Metric                     | Value                    |
+|----------------------------|--------------------------|
+| GPUs required              | 4-8x (H100/H200)        |
+| Framework                  | ms-swift + Megatron      |
+| Expert parallelism size    | 4 or 8                   |
+| Per-GPU VRAM               | ~40 GB with EP=8         |
+| System RAM per node        | 1+ TB recommended        |
+
+**ms-swift Megatron configuration for Qwen3.5-35B-A3B:**
+```bash
+NPROC_PER_NODE=8 swift sft \
+  --model Qwen/Qwen3.6-35B-A3B \
+  --train_type full \
+  --expert_model_parallel_size 8 \
+  --tuner_type lora --lora_rank 8 --lora_alpha 32 \
+  --moe_permute_fusion true \
+  --moe_grouped_gemm true \
+  --moe_shared_expert_overlap true \
+  --moe_aux_loss_coeff 1e-6 \
+  --offload_model true \
+  --offload_optimizer true
+```
+
+**Cost:** 8x H200 SXM on RunPod = ~$28.72/hr (community) or ~$35.12/hr (secure).
+This is expensive but may be necessary for true full-parameter training.
+
+**Warning:** Even 8x H200 with 1.5TB RAM has been reported to OOM on full SFT
+(ms-swift#6473). Expert parallelism with LoRA is the safer configuration.
+
+### Option 3: Selective Layer Unfreezing
+
+**Hybrid approach:** Freeze most experts, unfreeze shared expert + attention + embeddings.
+
+```python
+# Freeze all expert parameters
+for name, param in model.named_parameters():
+    if "experts" in name and "shared" not in name:
+        param.requires_grad = False
+
+# Keep trainable: attention, shared expert, embeddings, LM head
+# Trainable params: ~3-5B instead of 35B
+# Optimizer states: ~24-40 GB instead of 420 GB
+```
+
+This fits comfortably in 276 GB system RAM on 1x H200 and gives more capacity
+than LoRA while avoiding the full 35B optimizer state problem.
+
+### Option 4: Beast Server (NOT Recommended)
+
+The 3x RTX 3090 server has:
+- 72 GB total VRAM (3x 24 GB)
+- ~186 GB system RAM
+- Worse than 1x H200 in every dimension for this task
+
+Even with ZeRO-2 across 3 GPUs, the optimizer states for 35B params would need
+~420 GB CPU RAM. Does not fit. Would only work for LoRA, and the H200 is better
+for LoRA anyway (more VRAM per device, faster memory bandwidth).
+
+---
+
+## Diagnostic Checklist for the Running Pod
+
+If the pod is still accessible, run these to confirm the analysis:
+
+```bash
+# 1. Check ACTUAL container memory limit
+cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || \
+cat /sys/fs/cgroup/memory.max 2>/dev/null
+
+# 2. Compare with what free reports (will show host RAM, not limit)
+free -h
+
+# 3. Check OOM kill events
+dmesg | grep -i "oom\|killed\|memory" | tail -20
+
+# 4. Check cgroup memory usage at time of kill
+cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null || \
+cat /sys/fs/cgroup/memory.current 2>/dev/null
+
+# 5. Check if swap is available (usually no on RunPod)
+swapon --show
+
+# 6. Check GPU memory
+nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv
+```
+
+---
+
+## If You Still Want to Attempt Full SFT
+
+The absolute minimum requirements for full-parameter SFT of Qwen3.6-35B-A3B:
+
+1. **Multiple GPUs with Expert Parallelism** — not ZeRO-3 (unsupported for MoE)
+2. **At minimum 8 GPUs** with expert_parallel_size=8
+3. **System RAM:** 2+ TB to be safe (the ms-swift issue showed 1.5TB was not enough)
+4. **Framework:** ms-swift with Megatron backend, NOT raw DeepSpeed
+5. **Use LoRA even in multi-GPU** — full param SFT of 35B MoE may be fundamentally
+   impractical without custom expert-aware optimizer state partitioning
+6. **sub_group_size:** Set to 1e8 or lower (default 1e9) to reduce init memory peak
+7. **pin_memory: false** in offload config to reduce pinned memory allocation
+8. **Disable overlap_comm** during init if possible
+
+### DeepSpeed Config Tweaks (if retrying ZeRO-2 on multi-GPU)
+
+```json
+{
+  "zero_optimization": {
+    "stage": 2,
+    "offload_optimizer": {
+      "device": "cpu",
+      "pin_memory": false,
+      "buffer_count": 4,
+      "fast_init": false
+    },
+    "offload_param": {
+      "device": "cpu",
+      "pin_memory": false
+    },
+    "allgather_partitions": true,
+    "allgather_bucket_size": 5e7,
+    "overlap_comm": false,
+    "reduce_scatter": true,
+    "reduce_bucket_size": 5e7,
+    "contiguous_gradients": true,
+    "sub_group_size": 1e8
+  },
+  "bf16": { "enabled": true },
+  "train_micro_batch_size_per_gpu": 1,
+  "gradient_accumulation_steps": 4,
+  "gradient_clipping": 1.0
+}
+```
+
+Key changes from default:
+- `pin_memory: false` — pinned memory doubles the RAM footprint by keeping non-swappable copies
+- `sub_group_size: 1e8` — processes optimizer updates in smaller tiles, reducing peak init memory
+- `overlap_comm: false` — prevents double-buffering during init phase
+- `fast_init: false` — sequential initialization instead of bulk allocation
+- Reduced bucket sizes (5e7 vs default 5e8) — smaller communication buffers
+
+---
+
+## Decision Matrix
+
+| Approach                | Fits 1x H200? | Quality vs Full SFT | Cost/hr  | Complexity |
+|-------------------------|----------------|----------------------|----------|------------|
+| LoRA (Unsloth)          | YES            | ~95% for most tasks  | $3.59    | Low        |
+| Selective unfreeze      | YES            | ~97%                 | $3.59    | Medium     |
+| 8x H200 + EP + LoRA    | N/A (8 GPUs)   | ~98%                 | $28.72   | High       |
+| 8x H200 + EP + Full    | MAYBE          | 100% (if it works)   | $28.72   | Very High  |
+| Beast server            | NO             | N/A                  | Free     | N/A        |
+
+**Recommendation:** Use LoRA with Unsloth on the current 1x H200 pod. The quality
+difference between LoRA rank-16 and full SFT is minimal for SFT tasks, and the
+training will actually complete instead of being killed.
+
+---
+
+## Sources
+
+- [ms-swift#6473: System RAM OOM with Qwen3-30B-A3B](https://github.com/modelscope/ms-swift/issues/6473)
+- [DeepSpeed#2870: ZeRO-3 MoE unsupported](https://github.com/microsoft/DeepSpeed/issues/2870)
+- [DeepSpeed#7156: Expert optimizer state partitioning request](https://github.com/deepspeedai/DeepSpeed/issues/7156)
+- [DeepSpeed#2899: ZeRO-3 OOM with 30B model](https://github.com/microsoft/DeepSpeed/issues/2899)
+- [DeepSpeed#7021: CPU offload OOM with ZeRO-3](https://github.com/deepspeedai/DeepSpeed/issues/7021)
+- [DeepSpeed ZeRO-3 Documentation](https://deepspeed.readthedocs.io/en/latest/zero3.html)
+- [RunPod GPU Pricing (confirms 276 GB RAM per H200)](https://www.runpod.io/pricing)
+- [RunPod OOM Guide](https://www.runpod.io/articles/guides/avoid-oom-crashes-for-large-models)
+- [RunPod Resource Selection](https://www.runpod.io/blog/avoid-pod-errors-runpod-resources)
+- [Unsloth MoE Fine-tuning](https://unsloth.ai/docs/basics/faster-moe)
+- [Qwen3.6-35B-A3B Specs](https://apxml.com/models/qwen36-35b-a3b)
+- [Qwen3.5 MoE vs Dense Fine-tuning](https://medium.com/@ishaafsalman/qwen3-5-fine-tuning-in-2026-moe-vs-dense-b2d17de73a9e)
+- [ms-swift Qwen3.5 Best Practices](https://swift.readthedocs.io/en/latest/BestPractices/Qwen3_5-Best-Practice.html)
+- [Megatron Core MoE Training (arxiv:2603.07685)](https://arxiv.org/pdf/2603.07685)
+- [DeepSeek Memory Analysis (arxiv:2502.07846)](https://arxiv.org/pdf/2502.07846)