Daimon / specs /runpod_oom_debug.md

Add spec: runpod_oom_debug.md

82e4fd6 verified 11 days ago

14.5 kB

RunPod H200 SXM SIGKILL Debug: Qwen3.6-35B-A3B Full SFT

Date: 2026-06-30 Pod: vocal_coral_quokka (jifd0w19x1c61j) Status: Root cause identified. Full SFT is not viable on this pod configuration.

Root Cause: Container Memory Limit, Not Host RAM

The Lie `free -h` Tells You

RunPod containers see host RAM (2TB) in /proc/meminfo and free -h, but the actual memory available to the container is enforced by cgroup limits that are much lower. For a 1x H200 SXM pod, RunPod allocates:

Resource	Spec per 1x H200 SXM
VRAM	141 GB HBM3e
System RAM	276 GB
vCPUs	24

The container is capped at ~276 GB system RAM. When a process exceeds this, the Linux OOM killer sends SIGKILL (-9). There is no warning, no OOM error message, just instant death. This matches the observed behavior exactly.

How to Verify Inside the Pod

# cgroup v1 (most RunPod pods)
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# cgroup v2
cat /sys/fs/cgroup/memory.max

# What the process THINKS is available (WRONG - shows host RAM)
free -h

# What's actually enforced (check OOM events)
dmesg | grep -i "oom\|killed"

Memory Math: Why Full SFT Cannot Fit

Qwen3.6-35B-A3B Architecture

Total parameters: 35B (35 billion)
Active per token: 3B (8 routed + 1 shared expert of 256 total)
Layers: 40
Experts: 256 per MoE layer, 9 active
Hidden dim: 2,048
Architecture: Sparse MoE — inference is cheap, but training touches ALL 35B params

The Critical MoE Training Problem

During inference, only 3B parameters are active per token. But during training:

Forward pass: routes through selected experts (cheap)
Backward pass: computes gradients for ALL activated expert parameters
Optimizer step: maintains states for ALL 35B parameters regardless of activation

This means training memory scales with TOTAL params (35B), not active params (3B).

Memory Breakdown for Full SFT (Single GPU)

Component	Calculation	Size
Model weights (bf16)	35B x 2 bytes	70 GB
fp32 master weights	35B x 4 bytes	140 GB
AdamW momentum (fp32)	35B x 4 bytes	140 GB
AdamW variance (fp32)	35B x 4 bytes	140 GB
Gradients (bf16)	35B x 2 bytes	70 GB
Total (steady state)		560 GB
Activation memory	batch/seq dependent	10-50 GB+
CUDA/PyTorch overhead	fragmentation, buffers	5-15 GB
Peak during init	temporary copies	600-700 GB

What Each ZeRO Stage Actually Requires

ZeRO-2, no offload (attempt 1):

GPU needs: 70 (weights) + 140 (master) + 280 (optimizer) + 70 (grads) = 560 GB
Available GPU: 141 GB
Result: torch.OutOfMemoryError trying to allocate 129 GB. Correct behavior.

ZeRO-2, CPU offload (attempt 2):

GPU: ~70 GB model weights + activations + grad buffers ≈ 100-130 GB
CPU: fp32 master weights (140 GB) + optimizer states (280 GB) = 420 GB
Available CPU (container): 276 GB
Deficit: ~144 GB over limit
Result: SIGKILL during init. 100% CPU for 2 min = optimizer state allocation eating RAM until OOM killer fires.

ZeRO-3, CPU offload (attempt 3):

GPU: only active parameter slice + activations ≈ 20-40 GB
CPU: partitioned states BUT with 1 GPU, nothing to partition across
CPU needs: same ~420-560 GB for optimizer + master weights
Additionally: ZeRO-3 does NOT support MoE models. DeepSpeed raises AssertionError: MoE not supported with Stage 3 in current versions.
Even if it initialized, peak init memory creates temporary copies that spike higher
Result: SIGKILL after ~4 min of init. Same container RAM ceiling.

The ZeRO-3 + MoE Incompatibility

This is a confirmed, documented limitation:

GitHub Issue #2870 (filed Feb 2023, still open): "ZeRO stage 3 support for mixture-of-experts (MoE) layer" — explicitly unsupported
GitHub Issue #7156 (open): Even with ZeRO-2, expert optimizer states are NOT partitioned — only non-expert params get ZeRO treatment
The MoE expert parameters maintain full unpartitioned optimizer states, making them the dominant memory consumer

The ms-swift Confirmation

GitHub Issue modelscope/ms-swift#6473 documents the identical failure:

Same model family: Qwen3-30B-A3B (MoE)
Same setup: DeepSpeed ZeRO-2 with CPU offload
Same result: System RAM climbed to 1.7 TiB, OOM killed
Hardware: 8x H200 GPUs, 1.5TB RAM, 96 CPU cores
Key insight: Even with 8 GPUs and 1.5TB RAM, full SFT of this MoE model exhausted all system memory

If 8x H200s with 1.5TB RAM cannot do this, 1x H200 with 276 GB has zero chance.

Viable Alternatives (Ranked by Practicality)

Option 1: LoRA Fine-Tuning with Unsloth (RECOMMENDED)

Why it works: LoRA only trains adapter weights (~2-3% of params), so optimizer states are proportional to LoRA rank, not total model params.

Metric	Value
Trainable params	~931M of 36B (2.58%)
VRAM required (bf16 LoRA)	~63-74 GB
System RAM required	~40-80 GB
Fits on 1x H200 SXM?	Yes (141 GB VRAM)
Framework	Unsloth or ms-swift

Unsloth Configuration:

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Qwen/Qwen3.6-35B-A3B",
    max_seq_length=2048,        # Safe ceiling for 1 GPU
    dtype=torch.bfloat16,
    load_in_4bit=False,         # QLoRA NOT recommended for MoE (BnB limitation)
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,                       # LoRA rank
    lora_alpha=16,              # alpha == r, not r*2
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0,
    use_gradient_checkpointing="unsloth",
)

Critical Gotchas:

Set UNSLOTH_COMPILE_DISABLE=1 before imports (bf16/fp32 dtype mismatch in compiled MoE kernels)
Set dataloader_num_workers=0 and dataset_num_proc=1 (multiprocessing deadlock with large model)
Do NOT use QLoRA/4-bit — BitsandBytes does not support MoE architectures
Router layer is NOT fine-tuned by default (and shouldn't be for SFT)
Max safe sequence length on single H200: ~2048 tokens (activation memory explodes at 4096)
vLLM cannot load LoRA adapters directly — merge weights before serving

Unsloth Split LoRA provides additional MoE-specific optimization: reorders matmul to compute (X @ loraA) @ loraB instead of materializing loraA @ loraB, saving ~35% VRAM and providing ~2x speedup.

Option 2: Multi-GPU with Expert Parallelism (Megatron-SWIFT)

Why it works: Distributes expert params AND their optimizer states across GPUs. Each GPU only holds a subset of experts.

Metric	Value
GPUs required	4-8x (H100/H200)
Framework	ms-swift + Megatron
Expert parallelism size	4 or 8
Per-GPU VRAM	~40 GB with EP=8
System RAM per node	1+ TB recommended

ms-swift Megatron configuration for Qwen3.5-35B-A3B:

NPROC_PER_NODE=8 swift sft \
  --model Qwen/Qwen3.6-35B-A3B \
  --train_type full \
  --expert_model_parallel_size 8 \
  --tuner_type lora --lora_rank 8 --lora_alpha 32 \
  --moe_permute_fusion true \
  --moe_grouped_gemm true \
  --moe_shared_expert_overlap true \
  --moe_aux_loss_coeff 1e-6 \
  --offload_model true \
  --offload_optimizer true

Cost: 8x H200 SXM on RunPod = ~$28.72/hr (community) or ~$35.12/hr (secure). This is expensive but may be necessary for true full-parameter training.

Warning: Even 8x H200 with 1.5TB RAM has been reported to OOM on full SFT (ms-swift#6473). Expert parallelism with LoRA is the safer configuration.

Option 3: Selective Layer Unfreezing

Hybrid approach: Freeze most experts, unfreeze shared expert + attention + embeddings.

# Freeze all expert parameters
for name, param in model.named_parameters():
    if "experts" in name and "shared" not in name:
        param.requires_grad = False

# Keep trainable: attention, shared expert, embeddings, LM head
# Trainable params: ~3-5B instead of 35B
# Optimizer states: ~24-40 GB instead of 420 GB

This fits comfortably in 276 GB system RAM on 1x H200 and gives more capacity than LoRA while avoiding the full 35B optimizer state problem.

Option 4: Beast Server (NOT Recommended)

The 3x RTX 3090 server has:

72 GB total VRAM (3x 24 GB)
~186 GB system RAM
Worse than 1x H200 in every dimension for this task

Even with ZeRO-2 across 3 GPUs, the optimizer states for 35B params would need ~420 GB CPU RAM. Does not fit. Would only work for LoRA, and the H200 is better for LoRA anyway (more VRAM per device, faster memory bandwidth).

Diagnostic Checklist for the Running Pod

If the pod is still accessible, run these to confirm the analysis:

# 1. Check ACTUAL container memory limit
cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null || \
cat /sys/fs/cgroup/memory.max 2>/dev/null

# 2. Compare with what free reports (will show host RAM, not limit)
free -h

# 3. Check OOM kill events
dmesg | grep -i "oom\|killed\|memory" | tail -20

# 4. Check cgroup memory usage at time of kill
cat /sys/fs/cgroup/memory/memory.usage_in_bytes 2>/dev/null || \
cat /sys/fs/cgroup/memory.current 2>/dev/null

# 5. Check if swap is available (usually no on RunPod)
swapon --show

# 6. Check GPU memory
nvidia-smi --query-gpu=memory.total,memory.used,memory.free --format=csv

If You Still Want to Attempt Full SFT

The absolute minimum requirements for full-parameter SFT of Qwen3.6-35B-A3B:

Multiple GPUs with Expert Parallelism — not ZeRO-3 (unsupported for MoE)
At minimum 8 GPUs with expert_parallel_size=8
System RAM: 2+ TB to be safe (the ms-swift issue showed 1.5TB was not enough)
Framework: ms-swift with Megatron backend, NOT raw DeepSpeed
Use LoRA even in multi-GPU — full param SFT of 35B MoE may be fundamentally impractical without custom expert-aware optimizer state partitioning
sub_group_size: Set to 1e8 or lower (default 1e9) to reduce init memory peak
pin_memory: false in offload config to reduce pinned memory allocation
Disable overlap_comm during init if possible

DeepSpeed Config Tweaks (if retrying ZeRO-2 on multi-GPU)

{
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu",
      "pin_memory": false,
      "buffer_count": 4,
      "fast_init": false
    },
    "offload_param": {
      "device": "cpu",
      "pin_memory": false
    },
    "allgather_partitions": true,
    "allgather_bucket_size": 5e7,
    "overlap_comm": false,
    "reduce_scatter": true,
    "reduce_bucket_size": 5e7,
    "contiguous_gradients": true,
    "sub_group_size": 1e8
  },
  "bf16": { "enabled": true },
  "train_micro_batch_size_per_gpu": 1,
  "gradient_accumulation_steps": 4,
  "gradient_clipping": 1.0
}

Key changes from default:

pin_memory: false — pinned memory doubles the RAM footprint by keeping non-swappable copies
sub_group_size: 1e8 — processes optimizer updates in smaller tiles, reducing peak init memory
overlap_comm: false — prevents double-buffering during init phase
fast_init: false — sequential initialization instead of bulk allocation
Reduced bucket sizes (5e7 vs default 5e8) — smaller communication buffers

Decision Matrix

Approach	Fits 1x H200?	Quality vs Full SFT	Cost/hr	Complexity
LoRA (Unsloth)	YES	~95% for most tasks	$3.59	Low
Selective unfreeze	YES	~97%	$3.59	Medium
8x H200 + EP + LoRA	N/A (8 GPUs)	~98%	$28.72	High
8x H200 + EP + Full	MAYBE	100% (if it works)	$28.72	Very High
Beast server	NO	N/A	Free	N/A

Recommendation: Use LoRA with Unsloth on the current 1x H200 pod. The quality difference between LoRA rank-16 and full SFT is minimal for SFT tasks, and the training will actually complete instead of being killed.

LiberationLabs
/

Daimon

RunPod H200 SXM SIGKILL Debug: Qwen3.6-35B-A3B Full SFT

Root Cause: Container Memory Limit, Not Host RAM

The Lie `free -h` Tells You

How to Verify Inside the Pod

Memory Math: Why Full SFT Cannot Fit

Qwen3.6-35B-A3B Architecture

The Critical MoE Training Problem

Memory Breakdown for Full SFT (Single GPU)

What Each ZeRO Stage Actually Requires

The ZeRO-3 + MoE Incompatibility

The ms-swift Confirmation

Viable Alternatives (Ranked by Practicality)

Option 1: LoRA Fine-Tuning with Unsloth (RECOMMENDED)

Option 2: Multi-GPU with Expert Parallelism (Megatron-SWIFT)

Option 3: Selective Layer Unfreezing

Option 4: Beast Server (NOT Recommended)

Diagnostic Checklist for the Running Pod

If You Still Want to Attempt Full SFT

DeepSpeed Config Tweaks (if retrying ZeRO-2 on multi-GPU)

Decision Matrix

Sources

RunPod H200 SXM SIGKILL Debug: Qwen3.6-35B-A3B Full SFT

Root Cause: Container Memory Limit, Not Host RAM

The Lie free -h Tells You

How to Verify Inside the Pod

Memory Math: Why Full SFT Cannot Fit

Qwen3.6-35B-A3B Architecture

The Critical MoE Training Problem

Memory Breakdown for Full SFT (Single GPU)

What Each ZeRO Stage Actually Requires

The ZeRO-3 + MoE Incompatibility

The ms-swift Confirmation

Viable Alternatives (Ranked by Practicality)

Option 1: LoRA Fine-Tuning with Unsloth (RECOMMENDED)

Option 2: Multi-GPU with Expert Parallelism (Megatron-SWIFT)

Option 3: Selective Layer Unfreezing

Option 4: Beast Server (NOT Recommended)

Diagnostic Checklist for the Running Pod

If You Still Want to Attempt Full SFT

DeepSpeed Config Tweaks (if retrying ZeRO-2 on multi-GPU)

Decision Matrix

Sources

The Lie `free -h` Tells You