training-template/README.md

Daimon Training Template — RunPod

Full-parameter SFT of Qwen3.6-35B-A3B (MoE) on a single H200 SXM 141GB GPU. All 35B parameters trained in bf16 precision using DeepSpeed ZeRO-2 with CPU-offloaded Adafactor optimizer.

No LoRA. No adapters. No frozen layers. No half measures.

Liberation Labs, June 2026.

Memory Budget

Full-parameter SFT on a 35B MoE model requires careful memory planning across GPU and CPU.

GPU Memory (H200 SXM — 141GB VRAM)

Component	Size	Location
Model parameters (bf16)	~70 GB	GPU
Activations (with gradient checkpointing)	~20 GB	GPU
GPU Total	~90 GB	of 141 GB

CPU Memory (System RAM — 188GB)

Component	Size	Location
Gradients (bf16, ZeRO-2 offload)	~70 GB	CPU
Adafactor optimizer states	~35 GB	CPU
CPU Total	~105 GB	of 188 GB

Why Adafactor, Not AdamW

AdamW stores two fp32 states per parameter (momentum and variance):

• 35B params x 4 bytes x 2 states = 280 GB
• System RAM available: 188 GB
• Does not fit. Not even close.

Adafactor uses factored second moments — approximately one state per parameter in mixed precision, bringing optimizer memory to ~35GB. This is the only viable optimizer for full SFT on a single node with 188GB system RAM.

Checkpoint Storage

Full model checkpoints are ~70GB each (entire model in bf16). With save_total_limit=3:

• Checkpoint space: ~210 GB
• Model cache: ~70 GB
• Training data: ~5-10 GB
• Recommended persistent volume: 400 GB

Hardware Requirements

Resource	Minimum	Recommended
GPU	1x H200 SXM 141GB	1x H200 SXM 141GB
System RAM	180 GB	200 GB
Disk (/workspace)	300 GB	400 GB

Why a single H200? The H200 SXM has 141GB VRAM — enough to hold the entire 35B model in bf16 (70GB) plus activations with gradient checkpointing (20GB). DeepSpeed ZeRO-2 offloads optimizer states and gradients to CPU RAM. This avoids multi-GPU coordination complexity while training all parameters.

System RAM is critical. Unlike LoRA (where optimizer states are tiny), full SFT offloads ~105GB to CPU. Pods with < 180GB system RAM will OOM during training.

Environment Variables

Set these in the RunPod pod template:

Variable	Required	Description
HF_TOKEN	Yes	HuggingFace token (model is gated)
DAIMON_CONFIG	No	Override config path (default: bundled YAML)
DAIMON_MODEL	No	Override model path/ID
DAIMON_OUTPUT	No	Override output directory

Step-by-Step

1. Provision the Pod

• Template: RunPod PyTorch 2.x (or any CUDA 12.x image)
• GPU: 1x H200 SXM 141GB
• Container disk: 50 GB (for OS + packages)
• Volume disk: 400 GB (persistent, mounted at /workspace)
• Set HF_TOKEN in environment variables
• Verify system RAM >= 180GB before starting

2. Upload Template Files

Copy this directory to /workspace/runpod-template/ on the pod:

# From your local machine:
scp -r runpod-template/ root@<pod-ip>:/workspace/

Or clone the repo:

cd /workspace && git clone <repo-url> && cp -r Daimonion/runpod-template /workspace/

3. Run Setup

export HF_TOKEN="hf_your_token_here"
bash /workspace/runpod-template/setup.sh

This installs dependencies (pinned versions including DeepSpeed), downloads the model (~70GB), downloads training data, and verifies the environment. Takes 15-30 minutes depending on network speed.

4. Run Validation Tests

python3 /workspace/runpod-template/test_template.py

All 9 tests should pass before training. Tests now check both GPU VRAM and system RAM, validate the DeepSpeed ZeRO-2 config, confirm no LoRA artifacts remain, and estimate memory for both GPU and CPU.

5. Launch Training

bash /workspace/runpod-template/launch.sh

Training runs via deepspeed --num_gpus=1 with ZeRO Stage 2 CPU offload. The DeepSpeed launcher manages process initialization and ZeRO optimizer wrapping.

6. Monitor

# GPU utilization and VRAM usage
watch -n 5 nvidia-smi

# Training logs
tail -f /workspace/daimon-sft/logs/training_*.log

# CPU memory (watch for offloaded optimizer pressure)
watch -n 10 free -g

Resuming After Pod Restart

If the pod is terminated (spot instance preemption, manual stop, etc.), checkpoints are saved on the persistent volume. To resume:

1. Start a new pod with the same persistent volume
2. Run setup.sh again (it skips already-downloaded files)
3. Run launch.sh — it automatically finds the latest checkpoint

The training script scans /workspace/daimon-sft/checkpoint-* and resumes from the most recent one.

Post-Training

After training completes:

1. Copy the full model: scp -r root@<pod-ip>:/workspace/daimon-sft/final ./daimon-full-model/
   • This is the complete trained model (~70GB), not just adapters
   • It can be loaded directly with AutoModelForCausalLM.from_pretrained()
2. Delete checkpoints: rm -rf /workspace/daimon-sft/checkpoint-*
   • Each checkpoint is ~70GB — free the space
3. Terminate the pod and delete the persistent volume if no longer needed

Cost Estimates

Prices as of June 2026 (RunPod community cloud):

Configuration	$/hour	Est. time	Est. total
1x H200 SXM 141GB	~$4.39	20-30 hrs	$88-132

Full SFT is slower per step than LoRA (all 35B parameters updated per step), but produces a standalone model that doesn't need adapter merging or base model inference.

File Inventory

File	Purpose
setup.sh	First-boot: install deps (pinned versions + DeepSpeed), download model & data
train_daimon.py	Main training script (SFTTrainer + Adafactor + DeepSpeed ZeRO-2)
train_daimon_config.yaml	All hyperparameters in one place
ds_config_zero2.json	DeepSpeed ZeRO Stage 2 config with CPU optimizer offload
launch.sh	Pre-flight checks + DeepSpeed launch command
test_template.py	Validation tests — GPU, CPU RAM, config, memory estimates
ds_config_zero3.json	(LEGACY) Old ZeRO-3 config, no longer used

Configuration

Edit train_daimon_config.yaml to change hyperparameters. Key settings:

• max_seq_length: 4096 — Reduced from 8192 for full SFT (activation memory scales with sequence length). Increase if needed, but monitor GPU OOM.
• learning_rate: 5e-6 — Much lower than LoRA's 2e-4. Full SFT updates all parameters including MoE routing gates; higher rates destabilize routing.
• optimizer: adafactor — Only viable optimizer. AdamW needs 280GB CPU RAM.
• deepspeed_config: ds_config_zero2.json — ZeRO-2 with CPU optimizer offload.
• per_device_train_batch_size: 1 — With gradient_accumulation_steps=8, effective batch is 8.
• save_steps: 500 — Full model checkpoints are ~70GB each. With save_total_limit=3, up to ~210GB checkpoint space.
• save_total_limit: 3 — Keep only 3 checkpoints to avoid filling the 400GB volume.
• model_revision — Pinned to a specific HuggingFace commit hash for supply-chain security.

Known Issues and Solutions

Issue	Cause	Fix
GPU OOM during training	Sequence too long or batch size > 1	Reduce max_seq_length, verify per_device_train_batch_size=1, verify gradient_checkpointing is on
CPU OOM (killed by OS)	System RAM < 180GB or other processes using RAM	Provision pod with >= 188GB RAM, kill unnecessary processes
Disk full	Full checkpoints are ~70GB each	Reduce save_total_limit, increase volume to 400GB
Silent sequence truncation	max_seq_length too low	Increase in config; script pre-splits long sequences
Training killed by agent	Background process interference	This template runs as isolated DeepSpeed process
Lost checkpoints	Saved to container disk (not volume)	All output goes to /workspace/ (persistent)
DeepSpeed optimizer conflict	Optimizer set in both DS config and script	DS config has NO optimizer block — Adafactor managed by training script