training-template/README.md

# Daimon Training Template — RunPod

Full-parameter SFT of Qwen3.6-35B-A3B (MoE) on a single H200 SXM 141GB GPU. All 35B parameters trained in bf16 precision using DeepSpeed ZeRO-2 with CPU-offloaded Adafactor optimizer.

No LoRA. No adapters. No frozen layers. No half measures.

Liberation Labs, June 2026.

## Memory Budget

Full-parameter SFT on a 35B MoE model requires careful memory planning across GPU and CPU.

### GPU Memory (H200 SXM — 141GB VRAM)

| Component | Size | Location |
|-----------|------|----------|
| Model parameters (bf16) | ~70 GB | GPU |
| Activations (with gradient checkpointing) | ~20 GB | GPU |
| **GPU Total** | **~90 GB** | **of 141 GB** |

### CPU Memory (System RAM — 188GB)

| Component | Size | Location |
|-----------|------|----------|
| Gradients (bf16, ZeRO-2 offload) | ~70 GB | CPU |
| Adafactor optimizer states | ~35 GB | CPU |
| **CPU Total** | **~105 GB** | **of 188 GB** |

### Why Adafactor, Not AdamW

AdamW stores two fp32 states per parameter (momentum and variance):
- 35B params x 4 bytes x 2 states = **280 GB**
- System RAM available: 188 GB
- **Does not fit.** Not even close.

Adafactor uses factored second moments — approximately one state per parameter in mixed precision, bringing optimizer memory to ~35GB. This is the only viable optimizer for full SFT on a single node with 188GB system RAM.

### Checkpoint Storage

Full model checkpoints are **~70GB each** (entire model in bf16). With `save_total_limit=3`:
- Checkpoint space: ~210 GB
- Model cache: ~70 GB
- Training data: ~5-10 GB
- **Recommended persistent volume: 400 GB**

## Hardware Requirements

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| GPU | 1x H200 SXM 141GB | 1x H200 SXM 141GB |
| System RAM | 180 GB | 200 GB |
| Disk (/workspace) | 300 GB | 400 GB |

**Why a single H200?** The H200 SXM has 141GB VRAM — enough to hold the entire 35B model in bf16 (~70GB) plus activations with gradient checkpointing (~20GB). DeepSpeed ZeRO-2 offloads optimizer states and gradients to CPU RAM. This avoids multi-GPU coordination complexity while training all parameters.

**System RAM is critical.** Unlike LoRA (where optimizer states are tiny), full SFT offloads ~105GB to CPU. Pods with < 180GB system RAM will OOM during training.

## Environment Variables

Set these in the RunPod pod template:

| Variable | Required | Description |
|----------|----------|-------------|
| `HF_TOKEN` | Yes | HuggingFace token (model is gated) |
| `DAIMON_CONFIG` | No | Override config path (default: bundled YAML) |
| `DAIMON_MODEL` | No | Override model path/ID |
| `DAIMON_OUTPUT` | No | Override output directory |

## Step-by-Step

### 1. Provision the Pod

- Template: `RunPod PyTorch 2.x` (or any CUDA 12.x image)
- GPU: `1x H200 SXM 141GB`
- Container disk: 50 GB (for OS + packages)
- Volume disk: **400 GB** (persistent, mounted at `/workspace`)
- Set `HF_TOKEN` in environment variables
- **Verify system RAM >= 180GB before starting**

### 2. Upload Template Files

Copy this directory to `/workspace/runpod-template/` on the pod:

```bash
# From your local machine:
scp -r runpod-template/ root@<pod-ip>:/workspace/
```

Or clone the repo:

```bash
cd /workspace && git clone <repo-url> && cp -r Daimonion/runpod-template /workspace/
```

### 3. Run Setup

```bash
export HF_TOKEN="hf_your_token_here"
bash /workspace/runpod-template/setup.sh
```

This installs dependencies (pinned versions including DeepSpeed), downloads the model (~70GB), downloads training data, and verifies the environment. Takes 15-30 minutes depending on network speed.

### 4. Run Validation Tests

```bash
python3 /workspace/runpod-template/test_template.py
```

All 9 tests should pass before training. Tests now check both GPU VRAM and system RAM, validate the DeepSpeed ZeRO-2 config, confirm no LoRA artifacts remain, and estimate memory for both GPU and CPU.

### 5. Launch Training

```bash
bash /workspace/runpod-template/launch.sh
```

Training runs via `deepspeed --num_gpus=1` with ZeRO Stage 2 CPU offload. The DeepSpeed launcher manages process initialization and ZeRO optimizer wrapping.

### 6. Monitor

```bash
# GPU utilization and VRAM usage
watch -n 5 nvidia-smi

# Training logs
tail -f /workspace/daimon-sft/logs/training_*.log

# CPU memory (watch for offloaded optimizer pressure)
watch -n 10 free -g
```

## Resuming After Pod Restart

If the pod is terminated (spot instance preemption, manual stop, etc.), checkpoints are saved on the persistent volume. To resume:

1. Start a new pod with the same persistent volume
2. Run setup.sh again (it skips already-downloaded files)
3. Run launch.sh — it automatically finds the latest checkpoint

The training script scans `/workspace/daimon-sft/checkpoint-*` and resumes from the most recent one.

## Post-Training

After training completes:

1. **Copy the full model:** `scp -r root@<pod-ip>:/workspace/daimon-sft/final ./daimon-full-model/`
   - This is the complete trained model (~70GB), not just adapters
   - It can be loaded directly with `AutoModelForCausalLM.from_pretrained()`
2. **Delete checkpoints:** `rm -rf /workspace/daimon-sft/checkpoint-*`
   - Each checkpoint is ~70GB — free the space
3. **Terminate the pod** and delete the persistent volume if no longer needed

## Cost Estimates

Prices as of June 2026 (RunPod community cloud):

| Configuration | $/hour | Est. time | Est. total |
|--------------|--------|-----------|------------|
| 1x H200 SXM 141GB | ~$4.39 | 20-30 hrs | $88-132 |

Full SFT is slower per step than LoRA (all 35B parameters updated per step), but produces a standalone model that doesn't need adapter merging or base model inference.

## File Inventory

| File | Purpose |
|------|---------|
| `setup.sh` | First-boot: install deps (pinned versions + DeepSpeed), download model & data |
| `train_daimon.py` | Main training script (SFTTrainer + Adafactor + DeepSpeed ZeRO-2) |
| `train_daimon_config.yaml` | All hyperparameters in one place |
| `ds_config_zero2.json` | DeepSpeed ZeRO Stage 2 config with CPU optimizer offload |
| `launch.sh` | Pre-flight checks + DeepSpeed launch command |
| `test_template.py` | Validation tests — GPU, CPU RAM, config, memory estimates |
| `ds_config_zero3.json` | **(LEGACY)** Old ZeRO-3 config, no longer used |

## Configuration

Edit `train_daimon_config.yaml` to change hyperparameters. Key settings:

- `max_seq_length: 4096` — Reduced from 8192 for full SFT (activation memory scales with sequence length). Increase if needed, but monitor GPU OOM.
- `learning_rate: 5e-6` — Much lower than LoRA's 2e-4. Full SFT updates all parameters including MoE routing gates; higher rates destabilize routing.
- `optimizer: adafactor` — Only viable optimizer. AdamW needs 280GB CPU RAM.
- `deepspeed_config: ds_config_zero2.json` — ZeRO-2 with CPU optimizer offload.
- `per_device_train_batch_size: 1` — With gradient_accumulation_steps=8, effective batch is 8.
- `save_steps: 500` — Full model checkpoints are ~70GB each. With save_total_limit=3, up to ~210GB checkpoint space.
- `save_total_limit: 3` — Keep only 3 checkpoints to avoid filling the 400GB volume.
- `model_revision` — Pinned to a specific HuggingFace commit hash for supply-chain security.

## Known Issues and Solutions

| Issue | Cause | Fix |
|-------|-------|-----|
| GPU OOM during training | Sequence too long or batch size > 1 | Reduce max_seq_length, verify per_device_train_batch_size=1, verify gradient_checkpointing is on |
| CPU OOM (killed by OS) | System RAM < 180GB or other processes using RAM | Provision pod with >= 188GB RAM, kill unnecessary processes |
| Disk full | Full checkpoints are ~70GB each | Reduce save_total_limit, increase volume to 400GB |
| Silent sequence truncation | max_seq_length too low | Increase in config; script pre-splits long sequences |
| Training killed by agent | Background process interference | This template runs as isolated DeepSpeed process |
| Lost checkpoints | Saved to container disk (not volume) | All output goes to /workspace/ (persistent) |
| DeepSpeed optimizer conflict | Optimizer set in both DS config and script | DS config has NO optimizer block — Adafactor managed by training script |