Add training template: README.md

training-template/README.md

@@ -0,0 +1,199 @@
+# Daimon Training Template — RunPod
+
+Full-parameter SFT of Qwen3.6-35B-A3B (MoE) on a single H200 SXM 141GB GPU. All 35B parameters trained in bf16 precision using DeepSpeed ZeRO-2 with CPU-offloaded Adafactor optimizer.
+
+No LoRA. No adapters. No frozen layers. No half measures.
+
+Liberation Labs, June 2026.
+
+## Memory Budget
+
+Full-parameter SFT on a 35B MoE model requires careful memory planning across GPU and CPU.
+
+### GPU Memory (H200 SXM — 141GB VRAM)
+
+| Component | Size | Location |
+|-----------|------|----------|
+| Model parameters (bf16) | ~70 GB | GPU |
+| Activations (with gradient checkpointing) | ~20 GB | GPU |
+| **GPU Total** | **~90 GB** | **of 141 GB** |
+
+### CPU Memory (System RAM — 188GB)
+
+| Component | Size | Location |
+|-----------|------|----------|
+| Gradients (bf16, ZeRO-2 offload) | ~70 GB | CPU |
+| Adafactor optimizer states | ~35 GB | CPU |
+| **CPU Total** | **~105 GB** | **of 188 GB** |
+
+### Why Adafactor, Not AdamW
+
+AdamW stores two fp32 states per parameter (momentum and variance):
+- 35B params x 4 bytes x 2 states = **280 GB**
+- System RAM available: 188 GB
+- **Does not fit.** Not even close.
+
+Adafactor uses factored second moments — approximately one state per parameter in mixed precision, bringing optimizer memory to ~35GB. This is the only viable optimizer for full SFT on a single node with 188GB system RAM.
+
+### Checkpoint Storage
+
+Full model checkpoints are **~70GB each** (entire model in bf16). With `save_total_limit=3`:
+- Checkpoint space: ~210 GB
+- Model cache: ~70 GB
+- Training data: ~5-10 GB
+- **Recommended persistent volume: 400 GB**
+
+## Hardware Requirements
+
+| Resource | Minimum | Recommended |
+|----------|---------|-------------|
+| GPU | 1x H200 SXM 141GB | 1x H200 SXM 141GB |
+| System RAM | 180 GB | 200 GB |
+| Disk (/workspace) | 300 GB | 400 GB |
+
+**Why a single H200?** The H200 SXM has 141GB VRAM — enough to hold the entire 35B model in bf16 (~70GB) plus activations with gradient checkpointing (~20GB). DeepSpeed ZeRO-2 offloads optimizer states and gradients to CPU RAM. This avoids multi-GPU coordination complexity while training all parameters.
+
+**System RAM is critical.** Unlike LoRA (where optimizer states are tiny), full SFT offloads ~105GB to CPU. Pods with < 180GB system RAM will OOM during training.
+
+## Environment Variables
+
+Set these in the RunPod pod template:
+
+| Variable | Required | Description |
+|----------|----------|-------------|
+| `HF_TOKEN` | Yes | HuggingFace token (model is gated) |
+| `DAIMON_CONFIG` | No | Override config path (default: bundled YAML) |
+| `DAIMON_MODEL` | No | Override model path/ID |
+| `DAIMON_OUTPUT` | No | Override output directory |
+
+## Step-by-Step
+
+### 1. Provision the Pod
+
+- Template: `RunPod PyTorch 2.x` (or any CUDA 12.x image)
+- GPU: `1x H200 SXM 141GB`
+- Container disk: 50 GB (for OS + packages)
+- Volume disk: **400 GB** (persistent, mounted at `/workspace`)
+- Set `HF_TOKEN` in environment variables
+- **Verify system RAM >= 180GB before starting**
+
+### 2. Upload Template Files
+
+Copy this directory to `/workspace/runpod-template/` on the pod:
+
+```bash
+# From your local machine:
+scp -r runpod-template/ root@<pod-ip>:/workspace/
+```
+
+Or clone the repo:
+
+```bash
+cd /workspace && git clone <repo-url> && cp -r Daimonion/runpod-template /workspace/
+```
+
+### 3. Run Setup
+
+```bash
+export HF_TOKEN="hf_your_token_here"
+bash /workspace/runpod-template/setup.sh
+```
+
+This installs dependencies (pinned versions including DeepSpeed), downloads the model (~70GB), downloads training data, and verifies the environment. Takes 15-30 minutes depending on network speed.
+
+### 4. Run Validation Tests
+
+```bash
+python3 /workspace/runpod-template/test_template.py
+```
+
+All 9 tests should pass before training. Tests now check both GPU VRAM and system RAM, validate the DeepSpeed ZeRO-2 config, confirm no LoRA artifacts remain, and estimate memory for both GPU and CPU.
+
+### 5. Launch Training
+
+```bash
+bash /workspace/runpod-template/launch.sh
+```
+
+Training runs via `deepspeed --num_gpus=1` with ZeRO Stage 2 CPU offload. The DeepSpeed launcher manages process initialization and ZeRO optimizer wrapping.
+
+### 6. Monitor
+
+```bash
+# GPU utilization and VRAM usage
+watch -n 5 nvidia-smi
+
+# Training logs
+tail -f /workspace/daimon-sft/logs/training_*.log
+
+# CPU memory (watch for offloaded optimizer pressure)
+watch -n 10 free -g
+```
+
+## Resuming After Pod Restart
+
+If the pod is terminated (spot instance preemption, manual stop, etc.), checkpoints are saved on the persistent volume. To resume:
+
+1. Start a new pod with the same persistent volume
+2. Run setup.sh again (it skips already-downloaded files)
+3. Run launch.sh — it automatically finds the latest checkpoint
+
+The training script scans `/workspace/daimon-sft/checkpoint-*` and resumes from the most recent one.
+
+## Post-Training
+
+After training completes:
+
+1. **Copy the full model:** `scp -r root@<pod-ip>:/workspace/daimon-sft/final ./daimon-full-model/`
+   - This is the complete trained model (~70GB), not just adapters
+   - It can be loaded directly with `AutoModelForCausalLM.from_pretrained()`
+2. **Delete checkpoints:** `rm -rf /workspace/daimon-sft/checkpoint-*`
+   - Each checkpoint is ~70GB — free the space
+3. **Terminate the pod** and delete the persistent volume if no longer needed
+
+## Cost Estimates
+
+Prices as of June 2026 (RunPod community cloud):
+
+| Configuration | $/hour | Est. time | Est. total |
+|--------------|--------|-----------|------------|
+| 1x H200 SXM 141GB | ~$4.39 | 20-30 hrs | $88-132 |
+
+Full SFT is slower per step than LoRA (all 35B parameters updated per step), but produces a standalone model that doesn't need adapter merging or base model inference.
+
+## File Inventory
+
+| File | Purpose |
+|------|---------|
+| `setup.sh` | First-boot: install deps (pinned versions + DeepSpeed), download model & data |
+| `train_daimon.py` | Main training script (SFTTrainer + Adafactor + DeepSpeed ZeRO-2) |
+| `train_daimon_config.yaml` | All hyperparameters in one place |
+| `ds_config_zero2.json` | DeepSpeed ZeRO Stage 2 config with CPU optimizer offload |
+| `launch.sh` | Pre-flight checks + DeepSpeed launch command |
+| `test_template.py` | Validation tests — GPU, CPU RAM, config, memory estimates |
+| `ds_config_zero3.json` | **(LEGACY)** Old ZeRO-3 config, no longer used |
+
+## Configuration
+
+Edit `train_daimon_config.yaml` to change hyperparameters. Key settings:
+
+- `max_seq_length: 4096` — Reduced from 8192 for full SFT (activation memory scales with sequence length). Increase if needed, but monitor GPU OOM.
+- `learning_rate: 5e-6` — Much lower than LoRA's 2e-4. Full SFT updates all parameters including MoE routing gates; higher rates destabilize routing.
+- `optimizer: adafactor` — Only viable optimizer. AdamW needs 280GB CPU RAM.
+- `deepspeed_config: ds_config_zero2.json` — ZeRO-2 with CPU optimizer offload.
+- `per_device_train_batch_size: 1` — With gradient_accumulation_steps=8, effective batch is 8.
+- `save_steps: 500` — Full model checkpoints are ~70GB each. With save_total_limit=3, up to ~210GB checkpoint space.
+- `save_total_limit: 3` — Keep only 3 checkpoints to avoid filling the 400GB volume.
+- `model_revision` — Pinned to a specific HuggingFace commit hash for supply-chain security.
+
+## Known Issues and Solutions
+
+| Issue | Cause | Fix |
+|-------|-------|-----|
+| GPU OOM during training | Sequence too long or batch size > 1 | Reduce max_seq_length, verify per_device_train_batch_size=1, verify gradient_checkpointing is on |
+| CPU OOM (killed by OS) | System RAM < 180GB or other processes using RAM | Provision pod with >= 188GB RAM, kill unnecessary processes |
+| Disk full | Full checkpoints are ~70GB each | Reduce save_total_limit, increase volume to 400GB |
+| Silent sequence truncation | max_seq_length too low | Increase in config; script pre-splits long sequences |
+| Training killed by agent | Background process interference | This template runs as isolated DeepSpeed process |
+| Lost checkpoints | Saved to container disk (not volume) | All output goes to /workspace/ (persistent) |
+| DeepSpeed optimizer conflict | Optimizer set in both DS config and script | DS config has NO optimizer block — Adafactor managed by training script |