specs/runpod_oom_debug.md

# RunPod H200 SXM SIGKILL Debug: Qwen3.6-35B-A3B Full SFT

**Date:** 2026-06-30
**Pod:** vocal_coral_quokka (jifd0w19x1c61j)
**Status:** Root cause identified. Full SFT is not viable on this pod configuration.

---

## Root Cause: Container Memory Limit, Not Host RAM

### The Lie `free -h` Tells You

RunPod containers see **host RAM** (2TB) in `/proc/meminfo` and `free -h`, but the
actual memory available to the container is enforced by **cgroup limits** that are
much lower. For a 1x H200 SXM pod, RunPod allocates:

| Resource        | Spec per 1x H200 SXM |
|-----------------|----------------------|
| VRAM            | 141 GB HBM3e         |
| **System RAM**  | **276 GB**           |
| vCPUs           | 24                   |

**The container is capped at ~276 GB system RAM.** When a process exceeds this,
the Linux OOM killer sends SIGKILL (-9). There is no warning, no OOM error message,
just instant death. This matches the observed behavior exactly.

### How to Verify Inside the Pod

```bash
# cgroup v1 (most RunPod pods)
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# cgroup v2
cat /sys/fs/cgroup/memory.max

# What the process THINKS is available (WRONG - shows host RAM)
free -h

# What's actually enforced (check OOM events)
dmesg | grep -i "oom\|killed"
```

---

## Memory Math: Why Full SFT Cannot Fit

### Qwen3.6-35B-A3B Architecture

- **Total parameters:** 35B (35 billion)
- **Active per token:** 3B (8 routed + 1 shared expert of 256 total)
- **Layers:** 40
- **Experts:** 256 per MoE layer, 9 active
- **Hidden dim:** 2,048
- **Architecture:** Sparse MoE — inference is cheap, but **training touches ALL 35B params**

### The Critical MoE Training Problem

During inference, only 3B parameters are active per token. But during **training**:
- **Forward pass:** routes through selected experts (cheap)
- **Backward pass:** computes gradients for ALL activated expert parameters
- **Optimizer step:** maintains states for **ALL 35B parameters** regardless of activation

This means training memory scales with TOTAL params (35B), not active params (3B).

### Memory Breakdown for Full SFT (Single GPU)

| Component                  | Calculation           | Size     |
|----------------------------|-----------------------|----------|
| Model weights (bf16)       | 35B x 2 bytes         | **70 GB**  |
| fp32 master weights        | 35B x 4 bytes         | **140 GB** |
| AdamW momentum (fp32)      | 35B x 4 bytes         | **140 GB** |
| AdamW variance (fp32)      | 35B x 4 bytes         | **140 GB** |
| Gradients (bf16)           | 35B x 2 bytes         | **70 GB**  |
| **Total (steady state)**   |                       | **560 GB** |
| Activation memory          | batch/seq dependent   | 10-50 GB+  |
| CUDA/PyTorch overhead      | fragmentation, buffers| 5-15 GB  |
| **Peak during init**       | temporary copies      | **600-700 GB** |

### What Each ZeRO Stage Actually Requires

**ZeRO-2, no offload (attempt 1):**
- GPU needs: 70 (weights) + 140 (master) + 280 (optimizer) + 70 (grads) = **560 GB**
- Available GPU: 141 GB
- Result: `torch.OutOfMemoryError` trying to allocate 129 GB. Correct behavior.

**ZeRO-2, CPU offload (attempt 2):**
- GPU: ~70 GB model weights + activations + grad buffers ≈ 100-130 GB
- CPU: fp32 master weights (140 GB) + optimizer states (280 GB) = **420 GB**
- Available CPU (container): **276 GB**
- Deficit: **~144 GB over limit**
- Result: SIGKILL during init. 100% CPU for 2 min = optimizer state allocation eating RAM until OOM killer fires.

**ZeRO-3, CPU offload (attempt 3):**
- GPU: only active parameter slice + activations ≈ 20-40 GB
- CPU: partitioned states BUT with 1 GPU, nothing to partition across
- CPU needs: same ~420-560 GB for optimizer + master weights
- **Additionally:** ZeRO-3 does NOT support MoE models. DeepSpeed raises `AssertionError: MoE not supported with Stage 3` in current versions.
- Even if it initialized, peak init memory creates temporary copies that spike higher
- Result: SIGKILL after ~4 min of init. Same container RAM ceiling.

### The ZeRO-3 + MoE Incompatibility

This is a confirmed, documented limitation:
- **GitHub Issue #2870** (filed Feb 2023, still open): "ZeRO stage 3 support for mixture-of-experts (MoE) layer" — explicitly unsupported
- **GitHub Issue #7156** (open): Even with ZeRO-2, expert optimizer states are NOT partitioned — only non-expert params get ZeRO treatment
- The MoE expert parameters maintain full unpartitioned optimizer states, making them the dominant memory consumer

### The ms-swift Confirmation

GitHub Issue **modelscope/ms-swift#6473** documents the identical failure:
- **Same model family:** Qwen3-30B-A3B (MoE)  
- **Same setup:** DeepSpeed ZeRO-2 with CPU offload
- **Same result:** System RAM climbed to 1.7 TiB, OOM killed
- **Hardware:** 8x H200 GPUs, 1.5TB RAM, 96 CPU cores
- **Key insight:** Even with 8 GPUs and 1.5TB RAM, full SFT of this MoE model exhausted all system memory

If 8x H200s with 1.5TB RAM cannot do this, 1x H200 with 276 GB has zero chance.

---

## Viable Alternatives (Ranked by Practicality)

### Option 1: LoRA Fine-Tuning with Unsloth (RECOMMENDED)

**Why it works:** LoRA only trains adapter weights (~2-3% of params), so optimizer states
are proportional to LoRA rank, not total model params.

| Metric                     | Value                  |
|----------------------------|------------------------|
| Trainable params           | ~931M of 36B (2.58%)   |
| VRAM required (bf16 LoRA)  | ~63-74 GB              |
| System RAM required        | ~40-80 GB              |
| Fits on 1x H200 SXM?      | **Yes** (141 GB VRAM)  |
| Framework                  | Unsloth or ms-swift    |

**Unsloth Configuration:**
```python
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3.6-35B-A3B",
    max_seq_length=2048,        # Safe ceiling for 1 GPU
    dtype=torch.bfloat16,
    load_in_4bit=False,         # QLoRA NOT recommended for MoE (BnB limitation)
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,                       # LoRA rank
    lora_alpha=16,              # alpha == r, not r*2
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0,
    use_gradient_checkpointing="unsloth",
)
```

**Critical Gotchas:**
1. Set `UNSLOTH_COMPILE_DISABLE=1` before imports (bf16/fp32 dtype mismatch in compiled MoE kernels)
2. Set `dataloader_num_workers=0` and `dataset_num_proc=1` (multiprocessing deadlock with large model)
3. Do NOT use QLoRA/4-bit — BitsandBytes does not support MoE architectures
4. Router layer is NOT fine-tuned by default (and shouldn't be for SFT)
5. Max safe sequence length on single H200: ~2048 tokens (activation memory explodes at 4096)
6. vLLM cannot load LoRA adapters directly — merge weights before serving

**Unsloth Split LoRA** provides additional MoE-specific optimization: reorders matmul
to compute `(X @ loraA) @ loraB` instead of materializing `loraA @ loraB`, saving ~35%
VRAM and providing ~2x speedup.

### Option 2: Multi-GPU with Expert Parallelism (Megatron-SWIFT)

**Why it works:** Distributes expert params AND their optimizer states across GPUs.
Each GPU only holds a subset of experts.

| Metric                     | Value                    |
|----------------------------|--------------------------|
| GPUs required              | 4-8x (H100/H200)        |
| Framework                  | ms-swift + Megatron      |
| Expert parallelism size    | 4 or 8                   |
| Per-GPU VRAM               | ~40 GB with EP=8         |
| System RAM per node        | 1+ TB recommended        |

**ms-swift Megatron configuration for Qwen3.5-35B-A3B:**
```bash
NPROC_PER_NODE=8 swift sft \
  --model Qwen/Qwen3.6-35B-A3B \
  --train_type full \
  --expert_model_parallel_size 8 \
  --tuner_type lora --lora_rank 8 --lora_alpha 32 \
  --moe_permute_fusion true \
  --moe_grouped_gemm true \
  --moe_shared_expert_overlap true \
  --moe_aux_loss_coeff 1e-6 \
  --offload_model true \
  --offload_optimizer true
```

**Cost:** 8x H200 SXM on RunPod = ~$28.72/hr (community) or ~$35.12/hr (secure).
This is expensive but may be necessary for true full-parameter training.

**Warning:** Even 8x H200 with 1.5TB RAM has been reported to OOM on full SFT
(ms-swift#6473). Expert parallelism with LoRA is the safer configuration.

### Option 3: Selective Layer Unfreezing

**Hybrid approach:** Freeze most experts, unfreeze shared expert + attention + embeddings.

```python
# Freeze all expert parameters
for name, param in model.named_parameters():
    if "experts" in name and "shared" not in name:
        param.requires_grad = False

# Keep trainable: attention, shared expert, embeddings, LM head
# Trainable params: ~3-5B instead of 35B
# Optimizer states: ~24-40 GB instead of 420 GB
```

This fits comfortably in 276 GB system RAM on 1x H200 and gives more capacity
than LoRA while avoiding the full 35B optimizer state problem.

### Option 4: Beast Server (NOT Recommended)

The 3x RTX 3090 server has:
- 72 GB total VRAM (3x 24 GB)
- ~186 GB system RAM
- Worse than 1x H200 in every dimension for this task

Even with ZeRO-2 across 3 GPUs, the optimizer states for 35B params would need
~420 GB CPU RAM. Does not fit. Would only work for LoRA, and the H200 is better
for LoRA anyway (more VRAM per device, faster memory bandwidth).

---

## Diagnostic Checklist for the Running Pod

If the pod is still accessible, run these to confirm the analysis:

```bash
# 1. Check ACTUAL container memory limit
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || \
cat /sys/fs/cgroup/memory.max 2>/dev/null

# 2. Compare with what free reports (will show host RAM, not limit)
free -h

# 3. Check OOM kill events
dmesg | grep -i "oom\|killed\|memory" | tail -20

# 4. Check cgroup memory usage at time of kill
cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null || \
cat /sys/fs/cgroup/memory.current 2>/dev/null

# 5. Check if swap is available (usually no on RunPod)
swapon --show

# 6. Check GPU memory
nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv
```

---

## If You Still Want to Attempt Full SFT

The absolute minimum requirements for full-parameter SFT of Qwen3.6-35B-A3B:

1. **Multiple GPUs with Expert Parallelism** — not ZeRO-3 (unsupported for MoE)
2. **At minimum 8 GPUs** with expert_parallel_size=8
3. **System RAM:** 2+ TB to be safe (the ms-swift issue showed 1.5TB was not enough)
4. **Framework:** ms-swift with Megatron backend, NOT raw DeepSpeed
5. **Use LoRA even in multi-GPU** — full param SFT of 35B MoE may be fundamentally
   impractical without custom expert-aware optimizer state partitioning
6. **sub_group_size:** Set to 1e8 or lower (default 1e9) to reduce init memory peak
7. **pin_memory: false** in offload config to reduce pinned memory allocation
8. **Disable overlap_comm** during init if possible

### DeepSpeed Config Tweaks (if retrying ZeRO-2 on multi-GPU)

```json
{
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": false,
      "buffer_count": 4,
      "fast_init": false
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": false
    },
    "allgather_partitions": true,
    "allgather_bucket_size": 5e7,
    "overlap_comm": false,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e7,
    "contiguous_gradients": true,
    "sub_group_size": 1e8
  },
  "bf16": { "enabled": true },
  "train_micro_batch_size_per_gpu": 1,
  "gradient_accumulation_steps": 4,
  "gradient_clipping": 1.0
}
```

Key changes from default:
- `pin_memory: false` — pinned memory doubles the RAM footprint by keeping non-swappable copies
- `sub_group_size: 1e8` — processes optimizer updates in smaller tiles, reducing peak init memory
- `overlap_comm: false` — prevents double-buffering during init phase
- `fast_init: false` — sequential initialization instead of bulk allocation
- Reduced bucket sizes (5e7 vs default 5e8) — smaller communication buffers

---

## Decision Matrix

| Approach                | Fits 1x H200? | Quality vs Full SFT | Cost/hr  | Complexity |
|-------------------------|----------------|----------------------|----------|------------|
| LoRA (Unsloth)          | YES            | ~95% for most tasks  | $3.59    | Low        |
| Selective unfreeze      | YES            | ~97%                 | $3.59    | Medium     |
| 8x H200 + EP + LoRA    | N/A (8 GPUs)   | ~98%                 | $28.72   | High       |
| 8x H200 + EP + Full    | MAYBE          | 100% (if it works)   | $28.72   | Very High  |
| Beast server            | NO             | N/A                  | Free     | N/A        |

**Recommendation:** Use LoRA with Unsloth on the current 1x H200 pod. The quality
difference between LoRA rank-16 and full SFT is minimal for SFT tasks, and the
training will actually complete instead of being killed.

---

## Sources

- [ms-swift#6473: System RAM OOM with Qwen3-30B-A3B](https://github.com/modelscope/ms-swift/issues/6473)
- [DeepSpeed#2870: ZeRO-3 MoE unsupported](https://github.com/microsoft/DeepSpeed/issues/2870)
- [DeepSpeed#7156: Expert optimizer state partitioning request](https://github.com/deepspeedai/DeepSpeed/issues/7156)
- [DeepSpeed#2899: ZeRO-3 OOM with 30B model](https://github.com/microsoft/DeepSpeed/issues/2899)
- [DeepSpeed#7021: CPU offload OOM with ZeRO-3](https://github.com/deepspeedai/DeepSpeed/issues/7021)
- [DeepSpeed ZeRO-3 Documentation](https://deepspeed.readthedocs.io/en/latest/zero3.html)
- [RunPod GPU Pricing (confirms 276 GB RAM per H200)](https://www.runpod.io/pricing)
- [RunPod OOM Guide](https://www.runpod.io/articles/guides/avoid-oom-crashes-for-large-models)
- [RunPod Resource Selection](https://www.runpod.io/blog/avoid-pod-errors-runpod-resources)
- [Unsloth MoE Fine-tuning](https://unsloth.ai/docs/basics/faster-moe)
- [Qwen3.6-35B-A3B Specs](https://apxml.com/models/qwen36-35b-a3b)
- [Qwen3.5 MoE vs Dense Fine-tuning](https://medium.com/@ishaafsalman/qwen3-5-fine-tuning-in-2026-moe-vs-dense-b2d17de73a9e)
- [ms-swift Qwen3.5 Best Practices](https://swift.readthedocs.io/en/latest/BestPractices/Qwen3_5-Best-Practice.html)
- [Megatron Core MoE Training (arxiv:2603.07685)](https://arxiv.org/pdf/2603.07685)
- [DeepSeek Memory Analysis (arxiv:2502.07846)](https://arxiv.org/pdf/2502.07846)