Add corrected Daimon training pipeline v2.1 specification

training-template/daimon_training_pipeline_v2.md

@@ -0,0 +1,2014 @@
+# Daimon Training Pipeline v2.0
+# Liberation Labs House Model -- Implementation-Ready Specification
+# July 2026
+
+---
+
+## Executive Summary
+
+This document specifies the complete training pipeline for Daimon, Liberation Labs' prosocial house model -- Thomas Edrington's professional voice rendered as a sovereign model on Qwen3.6-35B-A3B.
+
+**Method:** HELLoRA (Hot Expert Layer-Level LoRA) with MAN-scored expert profiling, two-stage curriculum, OGPSA personality capture, and optional LASER post-training (offline). <!-- RED-HAT FIX #1: LASER moved to optional per C1 review -->
+
+**Hardware:** Single H200 SXM (141 GB HBM3e, 276 GB container RAM).
+
+**Budget:** $97 total at ~$2/hr (NeevCloud or PrimeIntellect). Estimated ~$30-45 for primary run (realistic raw-PEFT throughput), ~$52-67 reserved for iteration. <!-- RED-HAT FIX #12: Corrected budget from $28-36/$61-69 to realistic PEFT speeds per C4 review -->
+
+**Why not the v1 spec:** The v1 pipeline (daimon_training_spec.md) assumed Apple Silicon on Margaret with mlx-tune. This pipeline targets cloud H200 for the initial training run, then exports to GGUF/GPTQ for local serving. The v1 also specified 8 stages with DPO and RL -- this pipeline consolidates into fewer stages to reduce inter-stage forgetting risk and fit the $97 budget.
+
+---
+
+## TABLE OF CONTENTS
+
+1. [Critical Constraints (Non-Negotiable)](#1-critical-constraints)
+2. [Method Selection Rationale](#2-method-selection-rationale)
+3. [Phase 0: Preflight Checklist](#3-phase-0-preflight)
+4. [Phase 1: Expert Profiling](#4-phase-1-expert-profiling)
+5. [Phase 2: Foundation Training](#5-phase-2-foundation-training)
+6. [Phase 3: Thomas Voice Calibration](#6-phase-3-thomas-voice-calibration)
+7. [Phase 4: Post-Training](#7-phase-4-post-training)
+8. [Phase 5: Export and Validation](#8-phase-5-export-and-validation)
+9. [Budget Allocation](#9-budget-allocation)
+10. [Rollback Plan](#10-rollback-plan)
+11. [Appendix A: Environment Setup Script](#appendix-a)
+12. [Appendix B: Expert Profiling Script](#appendix-b)
+13. [Appendix C: Training Script](#appendix-c)
+14. [Appendix D: LASER Post-Training Script](#appendix-d)
+
+---
+
+## 1. Critical Constraints (Non-Negotiable) {#1-critical-constraints}
+
+These constraints were discovered during the SOTA sweep (sota_training_sweep_202607.md) and the RunPod OOM debug session. Violating any of these will waste budget.
+
+### Hard Stops
+
+| Constraint | Why | Source |
+|---|---|---|
+| **No QLoRA / 4-bit quantization** | BitsAndBytes 4-bit introduces routing perturbations on Qwen3.5/3.6 MoE. Causes token misrouting and quality degradation. Unsloth docs explicitly warn against it. | Unsloth docs, EAQuant (arXiv:2506.13329) |
+| **No router fine-tuning** | Routers co-evolve with expert geometry during pretraining. Modifying routers without corresponding expert adjustment causes performance degradation. | Unsloth, ESFT, HELLoRA, DR-LoRA, arXiv:2605.12476 |
+| **No DeepSpeed ZeRO-3** | Breaks gradient flow when LoRA adapters interact with MoE routing. Single GPU does not need DeepSpeed anyway. | Community reports, ZeRO docs |
+| **Freeze shared experts** | Training shared params causes overfitting and catastrophic forgetting. ESFT ablation is definitive. 1 shared expert per layer on Qwen3.6 -- freeze all 40. | ESFT (arXiv:2407.01906) ablation study |
+| **Freeze embeddings and LM head** | Unless specifically training for new vocabulary. Default Unsloth behavior. | Standard practice |
+| **Container RAM is 276 GB, not 2 TB** | `free -h` shows host RAM on RunPod. Actual limit is cgroup-enforced. Check with `cat /sys/fs/cgroup/memory/memory.limit_in_bytes` (v1) or `cat /sys/fs/cgroup/memory.max` (v2). Exceeding this triggers SIGKILL with no error message. | runpod_oom_debug.md |
+
+### Required Environment Variables
+
+These MUST be set BEFORE any Python imports:
+
+```bash
+export UNSLOTH_COMPILE_DISABLE=1
+export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
+export TOKENIZERS_PARALLELISM=false
+export UNSLOTH_DISABLE_FAST_GENERATION=1  # fixes RuntimeError from torch.compile on MoE
+```
+
+### Required Software Versions
+
+<!-- RED-HAT FIX #9: Pinned exact versions per H5 review. Version floors (>=) resolve to
+     whatever shipped that morning; pinned versions ensure reproducibility. Install torch
+     FIRST, unsloth --no-deps LAST. See Appendix A for install order. -->
+
+| Package | Version (pinned) | Why |
+|---|---|---|
+| torch | 2.7.1+cu124 | bf16 MoE training support; install FIRST to anchor resolution |
+| transformers | 5.0.2 | Qwen3.5/3.6 architecture support |
+| peft | 0.15.2 | Module-name-level target_modules for selective expert LoRA |
+| trl | 0.18.1 | SFTTrainer/SFTConfig API stability |
+| datasets | 3.6.0 | Streaming, interleaving |
+| accelerate | 1.7.0 | Training orchestration |
+| unsloth | >= 0.1.47-beta | MoE-specific Triton kernels, split LoRA; install LAST with --no-deps |
+
+### Training Hyperparameter Constraints
+
+| Parameter | Value | Why |
+|---|---|---|
+| `dataloader_num_workers` | 0 | Prevents MoE deadlocks with multi-process data loading |
+| `dataset_num_proc` | 1 | Same deadlock prevention |
+| Auxiliary loss coefficient | 0.01 | Mitigates expert collapse during fine-tuning |
+
+---
+
+## 2. Method Selection Rationale {#2-method-selection-rationale}
+
+### Why HELLoRA over alternatives
+
+| Method | Quality | VRAM (est.) | Throughput | Code Available | Risk |
+|---|---|---|---|---|---|
+| **Standard LoRA (all experts)** | Baseline | ~74 GB | 1x | Unsloth | Low |
+| **ESFT (full-param hot experts)** | 98.4% of full SFT | ~70-75 GB | ~0.7x | DeepSeek GitHub | Medium -- needs adaptation for Qwen3.6 |
+| **HELLoRA (LoRA on hot experts)** | > standard LoRA | ~90-115 GB peak | 1.9x (Unsloth) / ~0.5-0.7x (raw PEFT) | Manual implementation | Low | <!-- RED-HAT FIX #6: VRAM corrected; throughput split by code path -->
+| **BAdam (block-coordinate)** | 98.6% of full SFT | ~95-110 GB | ~0.5x | GitHub | Medium -- longer wall-clock |
+| **LISA (random layer unfreezing)** | > LoRA on MT-Bench | ~85-95 GB | ~0.8x | GitHub | Medium -- untested on MoE |
+
+**HELLoRA wins on the combination that matters for this budget:**
+
+1. **Memory efficiency:** ~90-115 GB peak (model bf16 70-72 GB + adapters/optimizer 3-8 GB + activations 8-15 GB + CE logits ~5 GB + CUDA overhead 3-6 GB) leaves ~25-50 GB headroom on H200. batch_size=4 at seq_len=2048 fits safely. <!-- RED-HAT FIX #6: Corrected VRAM from 55 GB to 90-115 GB per H1 review; removed seq_len=4096 claim which contradicts OOM debug findings -->
+2. **Training throughput:** 1.9x over standard LoRA (arXiv:2605.18795). Directly translates to more training per dollar.
+3. **Quality:** Outperforms standard LoRA on benchmarks (OLMoE GSM8K: 29.49 vs 26.37; Mixtral HumanEval: 44.89 vs 39.99).
+4. **Implementation simplicity:** HELLoRA is standard LoRA with selective module targeting. No custom training loops, no model surgery. Works with Unsloth/PEFT by constructing the right `target_modules` list.
+5. **Validated on 47B MoE:** Tested on Mixtral-8x7B (47B total), which is the closest published result to Qwen3.6-35B-A3B.
+6. **Rollback-friendly:** LoRA checkpoints are ~500 MB-1 GB. Can checkpoint every 2000 steps and roll back to any point without re-spending.
+
+**Why not ESFT:** ESFT does full parameter fine-tuning of hot experts, which means full optimizer states (Adam m + v) for those experts. At 10% of 35B params, that is ~3.5B trainable params requiring ~56 GB of optimizer state alone. HELLoRA with r=32 on the same hot experts requires <1 GB of optimizer state. ESFT has higher quality ceiling but the throughput penalty (no Unsloth MoE kernel acceleration for full-param training) means fewer iterations within budget.
+
+**Why not BAdam:** Near-full-SFT quality (98.6%) but at ~95-110 GB VRAM and ~0.5x throughput. Would consume the full budget in a single training run with no room for iteration.
+
+### LoRA Configuration
+
+| Parameter | Value | Rationale |
+|---|---|---|
+| Rank (r) | 32 | Can afford higher rank because we're only targeting ~9 hot experts per layer instead of 256. HELLoRA paper uses r=32. |
+| Alpha | 64 | Alpha = 2r per Microsoft guidance and training_pipeline_best_practices.md. Critical for high-rank convergence. |
+| Dropout | 0.05 | Light regularization. Lower than standard 0.1 because hot-expert-only training is already parameter-efficient. |
+| Target modules | Hot expert FFN layers + attention projections (see Phase 1 output) | HELLoRA protocol: LoRA on hot experts + attention, cold experts frozen. |
+| DoRA | True | +1-4% accuracy over vanilla LoRA at 5-10% memory overhead (arXiv:2402.09353, ICML 2024). Drop-in replacement. |
+
+---
+
+## 3. Phase 0: Preflight Checklist {#3-phase-0-preflight}
+
+**Cost:** $0 (local) or ~$1 (first 30 min on cloud if needed)
+**Time:** 30-60 minutes
+**Purpose:** Verify everything works BEFORE the meter starts running
+
+### 0.1 Hardware Verification (on the cloud instance)
+
+```bash
+# GPU check
+nvidia-smi --query-gpu=name,memory.total,memory.free --format=csv,noheader
+
+# Expected: NVIDIA H200, 141287 MiB total
+
+# Container RAM -- the REAL number (NOT free -h)
+# Try cgroup v2 first, fall back to v1
+CONTAINER_RAM=$(cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null)
+echo "Container RAM limit: $(echo "$CONTAINER_RAM / 1024 / 1024 / 1024" | bc) GB"
+
+# Expected: >= 276 GB. If less, STOP. Resize pod.
+if [ "$CONTAINER_RAM" -lt 270000000000 ]; then
+    echo "FATAL: Container RAM < 270 GB. Abort."
+    exit 1
+fi
+
+# CUDA version
+nvcc --version  # Expect 12.x
+
+# Disk space (need ~400 GB for model + checkpoints + merged weights + exports + data)
+# RED-HAT FIX #5: Increased from 150 GB to 400 GB per C5 review.
+# Artifact breakdown: base bf16 70 + adapter checkpoints w/ optimizer ~18 +
+# merged bf16 70 + GGUF intermediate 70 + Q4_K_M 18 + Q5_K_M 22 + data/cache ~6 = ~346 GB peak
+df -h /workspace
+DISK_FREE_GB=$(df --output=avail /workspace | tail -1 | awk '{print int($1/1048576)}')
+if [ "$DISK_FREE_GB" -lt 400 ]; then
+    echo "FATAL: Disk space < 400 GB free (have ${DISK_FREE_GB} GB). Abort."
+    exit 1
+fi
+```
+
+### 0.2 Environment Setup
+
+Run the full setup script (Appendix A). Verify:
+
+```bash
+# After setup, verify critical packages
+python -c "
+import torch; print(f'PyTorch: {torch.__version__}, CUDA: {torch.cuda.is_available()}')
+import transformers; print(f'Transformers: {transformers.__version__}')
+import peft; print(f'PEFT: {peft.__version__}')
+import unsloth; print(f'Unsloth: {unsloth.__version__}')
+assert torch.cuda.get_device_properties(0).total_memory > 140e9, 'Not H200'
+print('All checks passed.')
+"
+```
+
+### 0.3 Model Download (Do This Before Meter if Possible)
+
+```bash
+# Download model weights (~70 GB). If pod has persistent storage, do this once.
+huggingface-cli download Qwen/Qwen3.6-35B-A3B --local-dir /workspace/models/Qwen3.6-35B-A3B
+```
+
+### 0.4 Data Upload
+
+Upload all training datasets to `/workspace/data/`:
+
+```
+/workspace/data/
+  sonnet_voice_distill/      # 226K Sonnet voice (Roman1111111 + Nitral-AI + Norquinal)
+  prosocial_dialogue/        # 165K AllenAI prosocial (CC-BY-4.0)
+  xlam_function_calling/     # 60K agentic tool use
+  hermes_agent_traces/       # 7.6K reasoning traces
+  code_review_feedback/      # 9.5K CodeUltraFeedback
+  truthfulqa/                # 817 TruthfulQA
+  thomas_voice_demos/        # daimon_voice_demonstrations.jsonl (32 pairs)
+```
+
+### 0.5 One-Step Smoke Test
+
+```python
+import os
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"
+
+import torch
+from unsloth import FastLanguageModel
+
+model, tokenizer = FastLanguageModel.from_pretrained(
+    "Qwen/Qwen3.6-35B-A3B",
+    max_seq_length=2048,
+    load_in_4bit=False,
+    dtype=torch.bfloat16,
+)
+
+# Verify model loads and generates
+inputs = tokenizer("The daemon watches.", return_tensors="pt").to("cuda")
+with torch.no_grad():
+    output = model.generate(**inputs, max_new_tokens=20)
+print(tokenizer.decode(output[0]))
+
+# Check VRAM after model load
+allocated = torch.cuda.memory_allocated() / 1e9
+print(f"Model loaded: {allocated:.1f} GB VRAM")
+# Expected: ~70-72 GB
+
+del model, tokenizer
+torch.cuda.empty_cache()
+print("Smoke test passed.")
+```
+
+### 0.5b Backward-Pass Smoke Test <!-- RED-HAT FIX #11: Added per M10 review — verifies gradients flow through expert adapters before committing budget -->
+
+Run this AFTER the smoke test above. It loads the model via the same code path as the actual training script (Appendix C), applies LoRA, and verifies a backward pass updates adapter weights. This catches C2 (silent no-op training), C3 (eval crash), and H2 (wrong module topology) for ~$0.50.
+
+```python
+import os
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"
+
+import torch
+import hashlib
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from peft import LoraConfig, get_peft_model
+
+model_path = "/workspace/models/Qwen3.6-35B-A3B"
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path, torch_dtype=torch.bfloat16, device_map="cuda"
+)
+
+# Apply a minimal LoRA to a few modules to verify the training code path
+# Use a small subset of target_modules to keep it fast
+test_targets = []
+for proj in ["q_proj", "k_proj"]:
+    test_targets.append(f"model.layers.0.self_attn.{proj}")
+# Add one expert module to verify expert targeting works
+test_targets.append("model.layers.0.mlp.experts.0.gate_proj")
+
+lora_config = LoraConfig(
+    r=8, lora_alpha=16, lora_dropout=0.0,
+    target_modules=test_targets,
+    bias="none", task_type="CAUSAL_LM",
+)
+model = get_peft_model(model, lora_config)
+
+# Verify trainable params include expert modules
+expert_lora_count = sum(
+    1 for name, p in model.named_parameters()
+    if "experts" in name and "lora" in name and p.requires_grad
+)
+assert expert_lora_count > 0, "FATAL: No LoRA adapters on expert layers!"
+
+# Hash a LoRA weight before backward pass
+lora_param = None
+for name, p in model.named_parameters():
+    if "lora" in name and p.requires_grad:
+        lora_param = (name, p)
+        break
+pre_hash = hashlib.sha256(lora_param[1].data.cpu().numpy().tobytes()).hexdigest()
+
+# Forward + backward on one batch
+inputs = tokenizer("The daemon watches the threshold.", return_tensors="pt").to("cuda")
+labels = inputs["input_ids"].clone()
+outputs = model(**inputs, labels=labels)
+loss = outputs.loss
+loss.backward()
+
+# Verify gradients exist
+assert lora_param[1].grad is not None, f"FATAL: No gradient on {lora_param[0]}"
+assert lora_param[1].grad.abs().sum() > 0, f"FATAL: Zero gradient on {lora_param[0]}"
+
+# Simulate one optimizer step and verify weight changed
+with torch.no_grad():
+    lora_param[1].data -= 0.01 * lora_param[1].grad
+post_hash = hashlib.sha256(lora_param[1].data.cpu().numpy().tobytes()).hexdigest()
+assert pre_hash != post_hash, "FATAL: LoRA weight unchanged after optimizer step!"
+
+print(f"Backward-pass smoke test PASSED. Loss: {loss.item():.4f}")
+print(f"Expert LoRA params: {expert_lora_count}")
+print(f"Weight hash changed: {pre_hash[:12]}... -> {post_hash[:12]}...")
+
+del model, tokenizer
+torch.cuda.empty_cache()
+```
+
+### 0.6 Preflight Gate
+
+Do NOT proceed to Phase 1 unless ALL of the following are true:
+
+- [ ] H200 detected with >= 140 GB VRAM
+- [ ] Container RAM >= 270 GB (cgroup check, not free -h)
+- [ ] All Python packages import successfully at correct versions
+- [ ] Model downloads and generates text
+- [ ] Model VRAM after load is ~70-72 GB (confirms bf16, not accidental quantization)
+- [ ] All training data files present and readable
+- [ ] Disk space >= 400 GB free <!-- RED-HAT FIX #5: Increased from 150 GB per C5 review -->
+- [ ] Environment variables are set
+- [ ] Backward-pass smoke test passed (Section 0.5b) <!-- RED-HAT FIX #11: Added per M10 review -->
+- [ ] Expert LoRA adapters confirmed on expert modules (backward-pass test)
+
+---
+
+## 4. Phase 1: Expert Profiling {#4-phase-1-expert-profiling}
+
+**Cost:** ~$1 (30 minutes)
+**VRAM:** ~72 GB (model inference only, no optimizer states)
+**Purpose:** Identify which experts activate most frequently for our training data. This determines WHERE we apply LoRA.
+
+### Method: ESFT-Style Profiling with MAN Scoring
+
+Per ESFT (arXiv:2407.01906), 32 samples (~131K tokens) is sufficient for stable expert profiling. We profile on a representative subset of our actual training data to identify the experts that will handle Daimon's workload.
+
+Per the Unified Expert Scoring Framework (arXiv:2606.15716, tested directly on Qwen3-30B-A3B), MAN (Mean Activation Norm) is the best scoring method, outperforming frequency, MSAN, REAP, and SEER.
+
+### Profiling Script
+
+See Appendix B for the complete script. The algorithm:
+
+1. Load model in bf16 (inference only, ~72 GB VRAM)
+2. Sample 32 examples from training data: 8 from Sonnet voice, 8 from prosocial, 8 from tool use, 4 from reasoning, 4 from Thomas demos
+3. For each example, hook every MoE layer's router
+4. Record: (a) which experts are selected by top-k routing, (b) the activation norm of each selected expert's output
+5. Score each expert per layer by Mean Activation Norm across all 32 samples
+6. Identify "hot" experts: top N experts per layer where cumulative MAN covers >= 80% of total activation norm
+7. Output: `hot_experts.json` mapping layer_idx -> list of hot expert indices
+
+### Expected Output
+
+Based on Qwen3.6's architecture (256 routed experts, top-8 routing), we expect:
+
+- ~20-40 hot experts per layer (8-16% of 256)
+- Strong skew: Qwen3.6 uses fine-grained experts with sharp routing, meaning a small subset handles most traffic for any given data distribution
+- Consistency across layers: some experts will be hot in most layers, others layer-specific
+
+### Profiling Gate
+
+- [ ] `hot_experts.json` generated successfully
+- [ ] Each layer has 15-50 hot experts (if <15 or >100, something is wrong with the profiling)
+- [ ] Activation norm distribution shows clear skew (top 10% of experts carry >50% of norm)
+- [ ] Save `hot_experts.json` to persistent storage -- this is the key artifact for Phase 2
+
+---
+
+## 5. Phase 2: Foundation Training {#5-phase-2-foundation-training}
+
+**Cost:** ~$22-32 (11-16 hours), gated by throughput probe <!-- RED-HAT FIX #4: Corrected from $20-28 to realistic raw-PEFT throughput per C4 review -->
+**VRAM:** ~90-115 GB peak (model bf16 70-72 GB + adapters/optimizer 3-8 GB + activations 8-15 GB + CE logits ~5 GB + CUDA overhead 3-6 GB) <!-- RED-HAT FIX #6: Corrected from 55-65 GB per H1 review -->
+**Purpose:** Train the complete Daimon capability and voice foundation in a single mixed-data pass
+
+### 5.1 Dataset Composition
+
+All training data is mixed into a single dataset with proportional sampling weights. Small datasets are upsampled to prevent being drowned out by large ones.
+
+| Dataset | Raw Count | Upsample Factor | Effective Count | Sampling Weight | Purpose |
+|---|---|---|---|---|---|
+| Sonnet voice distill | 226,000 | 1x | 226,000 | 0.44 | Core voice and linguistic quality |
+| Prosocial dialogue | 165,000 | 0.3x (subsample) | 49,500 | 0.10 | Values alignment, prosocial patterns |
+| xlam function calling | 60,000 | 1x | 60,000 | 0.12 | Tool use capability |
+| Hermes agent traces | 7,600 | 3x | 22,800 | 0.04 | Multi-step reasoning |
+| Code review feedback | 9,500 | 2x | 19,000 | 0.04 | Constructive feedback style |
+| TruthfulQA | 817 | 15x | 12,255 | 0.02 | Anti-sycophancy inoculation |
+| **Total effective** | | | **~390,000** | **1.00** | |
+
+**Why subsample prosocial to 30%:** 165K prosocial dialogue examples would dominate the voice signal from the 226K Sonnet distill. At 49.5K, it provides sufficient alignment signal without overwhelming the primary voice training data. The prosocial patterns are relatively simple (be helpful, don't be harmful) and converge faster than complex voice patterns.
+
+**Why upsample TruthfulQA 15x:** 817 examples in a 390K-sample dataset would be seen <1 time per epoch. At 15x, the model sees each TruthfulQA example ~15 times, providing sufficient anti-sycophancy signal. Overfitting risk on 817 unique examples is manageable at this upsample factor with LoRA.
+
+### 5.2 Data Format
+
+All data must be in Unsloth's chat template format. For Qwen3.6:
+
+```python
+# Each example as a list of messages
+{"messages": [
+    {"role": "system", "content": "You are Daimon, a strategic thinking partner..."},
+    {"role": "user", "content": "..."},
+    {"role": "assistant", "content": "..."}
+]}
+```
+
+**System prompt for voice distill data:** Use a minimal system prompt (or none) to let the Sonnet voice dominate. The voice demonstrations carry the persona; the system prompt is a runtime artifact, not a training signal.
+
+**System prompt for tool use data:** Include the tool definitions in the system prompt, matching the xlam format.
+
+**System prompt for Thomas voice demos:** No system prompt. The separation principle: how to be, not what to recite. The demos teach the model Thomas's voice through example, not instruction.
+
+### 5.3 LoRA Target Module Construction
+
+This is the core HELLoRA implementation. Using the `hot_experts.json` from Phase 1, construct the PEFT `target_modules` list:
+
+```python
+import json
+
+with open("hot_experts.json") as f:
+    hot_experts = json.load(f)
+
+# Build target module list for PEFT
+target_modules = []
+
+# Attention projections (all layers) -- secondary target per HELLoRA
+for layer_idx in range(40):
+    for proj in ["q_proj", "k_proj", "v_proj", "o_proj"]:
+        target_modules.append(f"model.layers.{layer_idx}.self_attn.{proj}")
+
+# Hot expert FFN layers (primary target per HELLoRA)
+for layer_idx_str, expert_indices in hot_experts.items():
+    layer_idx = int(layer_idx_str)
+    for expert_idx in expert_indices:
+        for proj in ["gate_proj", "up_proj", "down_proj"]:
+            target_modules.append(
+                f"model.layers.{layer_idx}.mlp.experts.{expert_idx}.{proj}"
+            )
+
+# Explicitly NOT included:
+# - Router weights (model.layers.*.mlp.gate.*)     -- FROZEN
+# - Shared expert (model.layers.*.mlp.shared_expert.*)  -- FROZEN
+# - Cold expert FFN layers                          -- FROZEN
+# - Embedding / LM head                             -- FROZEN
+
+print(f"Total target modules: {len(target_modules)}")
+# Expected: 160 attention modules + (hot_experts_per_layer * 40 layers * 3 projections)
+# If ~25 hot experts/layer: 160 + 25*40*3 = 160 + 3000 = 3160 modules
+```
+
+### 5.4 Training Configuration
+
+```python
+from unsloth import FastLanguageModel
+from trl import SFTTrainer, SFTConfig
+import torch
+
+# Load model
+model, tokenizer = FastLanguageModel.from_pretrained(
+    "Qwen/Qwen3.6-35B-A3B",
+    max_seq_length=2048,
+    load_in_4bit=False,  # CRITICAL: Must be False
+    dtype=torch.bfloat16,
+)
+
+# Apply HELLoRA with hot expert targeting
+# NOTE: If Unsloth's get_peft_model does not support explicit module name lists,
+# use PEFT directly:
+from peft import LoraConfig, get_peft_model
+
+lora_config = LoraConfig(
+    r=32,
+    lora_alpha=64,
+    lora_dropout=0.05,
+    target_modules=target_modules,  # From Section 5.3
+    use_dora=True,                  # DoRA upgrade (+1-4% accuracy)
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+
+model = get_peft_model(model, lora_config)
+
+# VERIFY trainable params include expert modules
+trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
+total_params = sum(p.numel() for p in model.parameters())
+print(f"Trainable: {trainable_params:,} ({100*trainable_params/total_params:.2f}%)")
+# Expected: 0.3-0.8% of total params (per HELLoRA paper)
+# If >3%, something is wrong -- too many modules are unfrozen
+# If <0.1%, hot expert targeting is too aggressive -- expand threshold
+
+# CRITICAL CHECK (mlx-lm bug #571 equivalent for PEFT):
+# Verify that expert modules actually have LoRA adapters
+expert_lora_count = sum(1 for name, _ in model.named_parameters()
+                        if "experts" in name and "lora" in name and _.requires_grad)
+print(f"Expert LoRA params: {expert_lora_count}")
+assert expert_lora_count > 0, "FATAL: No LoRA adapters on expert layers!"
+
+# RED-HAT FIX #3: Create stratified eval split BEFORE interleaving training data (per C3 review).
+# Without this, eval_strategy="steps" crashes at init or first eval step.
+# ~250-300 examples held out, stratified across all 6 sources, fixed seed, excluded from training.
+# See build_foundation_dataset() in Appendix C for implementation.
+
+# Training config
+training_args = SFTConfig(
+    output_dir="/workspace/checkpoints/daimon_v2_foundation",
+    per_device_train_batch_size=4,       # 90-115 GB peak leaves room for batch=4 on H200
+    gradient_accumulation_steps=4,       # Effective batch = 16
+    num_train_epochs=1,                  # Single epoch for ~390K examples
+    learning_rate=2e-4,
+    lr_scheduler_type="cosine",
+    warmup_ratio=0.03,                   # ~750 warmup steps
+    weight_decay=0.01,
+    bf16=True,
+    max_seq_length=2048,
+    logging_steps=50,
+    save_steps=2000,                     # Checkpoint every 2000 steps (~$2 of training)
+    save_total_limit=5,                  # Keep last 5 checkpoints to manage disk
+    dataloader_num_workers=0,            # CRITICAL: Prevents MoE deadlocks
+    dataset_num_proc=1,                  # CRITICAL: Prevents MoE deadlocks
+    gradient_checkpointing=True,         # Required for memory safety
+    gradient_checkpointing_kwargs={"use_reentrant": False},
+    eval_strategy="steps",
+    eval_steps=2000,                     # Eval at every checkpoint
+    load_best_model_at_end=True,         # RED-HAT FIX #3: Select best checkpoint by eval_loss (per C3 review)
+    metric_for_best_model="eval_loss",   # RED-HAT FIX #3: Best = lowest eval loss
+    max_steps=30000,                     # RED-HAT FIX #4: Circuit breaker — hard cap to prevent budget overrun (per C4 review). Adjusted after throughput probe.
+    report_to="none",                    # Or "wandb" if experiment tracking is set up
+    seed=42,
+)
+```
+
+### 5.5 Training Time Estimate
+
+<!-- RED-HAT FIX #4: Replaced Unsloth kernel speeds with realistic raw-PEFT speeds per C4 review.
+     The shipped script uses raw PEFT + Transformers, not Unsloth MoE kernels.
+     Unsloth throughput (1.5-2.0 s/step) is only achievable if FastLanguageModel.get_peft_model()
+     supports explicit module-name lists. Until verified, budget for the slower path. -->
+
+| Parameter | Value |
+|---|---|
+| Effective samples | ~390,000 |
+| Effective batch size | 16 |
+| Steps per epoch | ~24,375 |
+| Estimated sec/step (raw PEFT, no Unsloth kernels) | 2.5-4.0 |
+| Wall-clock estimate | 17-27 hours |
+| Cost at $2/hr | $34-54 (but gated by throughput probe — see 5.5b) |
+| Realistic budget after probe gate | $22-32 (probe rejects runs above this) |
+
+**Note:** If Unsloth's `FastLanguageModel.get_peft_model()` accepts explicit module-name lists (verify during preflight), throughput improves to ~1.5-2.0 s/step and cost drops to ~$20-28. The probe (Section 5.5b) measures the actual throughput and gates accordingly.
+
+### 5.5b Throughput Probe (Hard Gate) <!-- RED-HAT FIX #4 + RED-HAT FIX #10: Added per C4 review -->
+
+**Cost:** ~$0.50-1.00 (50 steps)
+**Purpose:** Measure actual tokens/sec before committing budget. This is the single most valuable addition to the pipeline.
+
+Run 50 training steps on the real data mix with the real config. Measure tokens/sec and project Phase 2 cost.
+
+```python
+# After model + LoRA setup, before full training:
+import time
+
+# Run 50 steps as a probe
+probe_args = SFTConfig(
+    output_dir="/workspace/checkpoints/daimon_v2_probe",
+    per_device_train_batch_size=4,
+    gradient_accumulation_steps=4,
+    max_steps=50,
+    learning_rate=2e-4,
+    bf16=True,
+    max_seq_length=2048,
+    logging_steps=10,
+    save_strategy="no",
+    dataloader_num_workers=0,
+    dataset_num_proc=1,
+    gradient_checkpointing=True,
+    gradient_checkpointing_kwargs={"use_reentrant": False},
+    report_to="none",
+    seed=42,
+)
+
+probe_trainer = SFTTrainer(
+    model=model,
+    args=probe_args,
+    train_dataset=train_dataset,
+    processing_class=tokenizer,
+)
+
+start_time = time.time()
+probe_trainer.train()
+elapsed = time.time() - start_time
+
+secs_per_step = elapsed / 50
+tokens_per_sec = (4 * 4 * 2048) / secs_per_step  # batch * accum * seq_len
+projected_hours = (24375 * secs_per_step) / 3600
+projected_cost = projected_hours * 2.0  # at $2/hr
+
+print(f"Throughput probe results:")
+print(f"  {secs_per_step:.2f} sec/step")
+print(f"  {tokens_per_sec:.0f} tokens/sec")
+print(f"  Projected Phase 2: {projected_hours:.1f} hours, ${projected_cost:.0f}")
+
+# HARD GATE: abort if projected cost exceeds budget cap
+COST_CAP = 35.0  # Max dollars for Phase 2
+if projected_cost > COST_CAP:
+    print(f"ABORT: Projected cost ${projected_cost:.0f} exceeds cap ${COST_CAP:.0f}.")
+    print(f"Fallback: Switch to Tier-1 Unsloth all-expert LoRA r=16.")
+    raise SystemExit(1)
+
+# Set max_steps based on probe to enforce cost ceiling
+safe_max_steps = int(COST_CAP / 2.0 * 3600 / secs_per_step)
+print(f"  Setting max_steps={safe_max_steps} as cost circuit breaker")
+```
+
+**Fallback if probe fails (projected cost > $35):** Switch to Tier-1 Unsloth-native all-expert LoRA with r=16, alpha=16 -- the one configuration with measured throughput on this model family.
+
+### 5.6 Monitoring During Training
+
+Watch for these during Phase 2:
+
+```python
+# Add to training callbacks or check manually between checkpoints:
+
+# 1. Loss curve -- should decrease smoothly
+#    Red flag: sudden spike or plateau before step 5000
+
+# 2. VRAM usage -- should be stable
+#    Red flag: gradual increase (memory leak from activation caching)
+
+# 3. Learning rate -- should follow cosine schedule
+#    Red flag: NaN or zero (optimizer failure)
+```
+
+Monitor expert utilization at eval steps (add as callback):
+
+```python
+# Quick check: are hot experts still hot?
+# If routing has shifted significantly, the LoRA placement is misaligned.
+# This is informational -- do not retrain Phase 1 mid-run.
+```
+
+### 5.6b Checkpoint Egress <!-- RED-HAT FIX #8: Added per H9 review — checkpoints must leave the pod after each save -->
+
+**Spot instances can terminate at any time.** Checkpoints only on the pod volume are not insurance -- they are gone when the volume is reclaimed. After every checkpoint save, rsync the adapter directory to persistent off-pod storage.
+
+```bash
+# Add as a post-checkpoint callback or cron job running every 30 minutes:
+# Adapter checkpoint is ~3-4 GB (includes optimizer state for resuming)
+
+REMOTE_DEST="user@margaret.local:/workspace/daimon_checkpoints/"  # or MTH, or B2 bucket
+CHECKPOINT_DIR="/workspace/checkpoints/daimon_v2_foundation"
+
+# Sync latest checkpoint off-pod after each save
+rsync -avz --progress \
+    "${CHECKPOINT_DIR}/$(ls -td ${CHECKPOINT_DIR}/checkpoint-* | head -1)" \
+    "${REMOTE_DEST}"
+
+# Also sync critical artifacts
+rsync -avz /workspace/artifacts/hot_experts.json "${REMOTE_DEST}"
+```
+
+**Cost:** Pennies of bandwidth. Converts "total loss on spot preemption" into "resume from last checkpoint - 2000 steps."
+
+Add inter-phase disk check:
+```bash
+# Run between phases to catch disk pressure before it corrupts a save
+DISK_FREE_GB=$(df --output=avail /workspace | tail -1 | awk '{print int($1/1048576)}')
+echo "Disk free: ${DISK_FREE_GB} GB"
+if [ "$DISK_FREE_GB" -lt 80 ]; then
+    echo "WARNING: Disk below 80 GB free. Clean stale checkpoints before continuing."
+fi
+```
+
+### 5.7 Phase 2 Gate
+
+Before proceeding to Phase 3:
+
+- [ ] Training loss decreased smoothly (no divergence, no NaN)
+- [ ] Final training loss < initial loss by at least 30%
+- [ ] Validation loss is within 10% of training loss (no severe overfitting)
+- [ ] Generate 5 test prompts manually and verify coherent output
+- [ ] VRAM stayed within expected bounds (~90-115 GB peak, alarm at >125 GB sustained) <!-- RED-HAT FIX #6: Corrected from 55-65 GB per H1 review -->
+- [ ] All 5 most recent checkpoints saved successfully
+- [ ] Best checkpoint selected by `load_best_model_at_end=True` via eval_loss <!-- RED-HAT FIX #3: Auto-selected, not manual -->
+- [ ] Best checkpoint copied to `/workspace/checkpoints/daimon_v2_foundation_best/`
+- [ ] Best checkpoint synced off-pod to persistent storage <!-- RED-HAT FIX #8: Per H9 review -->
+
+---
+
+## 6. Phase 3: Thomas Voice Calibration {#6-phase-3-thomas-voice-calibration}
+
+**Cost:** ~$1 (30 minutes)
+**VRAM:** ~90-115 GB peak (same as Phase 2) <!-- RED-HAT FIX #6: Corrected from 55-65 GB per H1 review -->
+**Purpose:** Final persona imprint. The Thomas voice demonstrations teach the model HOW to be Daimon -- the specific patterns of question-before-answer, metaphor-forward explanation, constructive challenge, dry wit. This is the separation principle: procedural knowledge, not declarative content.
+
+### 6.1 Data
+
+Source: `/home/HumboldtJoker/.coalition/specs/daimon_voice_demonstrations.jsonl`
+Count: 32 procedural demonstrations (from the existing v1 spec, already authored)
+Additional: If available, 8-10 more demonstrations covering edge cases (overwhelming user, technical disagreement, values-laden questions). Target 40 total.
+
+Format: Already in `{"messages": [...]}` format. No system prompt -- pure procedural demonstration.
+
+### 6.2 Training Configuration
+
+```python
+# RED-HAT FIX #2: Phase 2 saves adapter-only (adapter_config.json + adapter_model.safetensors),
+# NOT a full model. Must load base model first, then attach adapter with is_trainable=True.
+# Using AutoModelForCausalLM.from_pretrained on an adapter dir will either crash
+# ("no config.json") or load adapter for inference-only (requires_grad=False),
+# causing silent no-op training. (Per C2 review)
+
+from peft import PeftModel
+
+# Load base model first
+base_model = AutoModelForCausalLM.from_pretrained(
+    "/workspace/models/Qwen3.6-35B-A3B",  # Base model, NOT the checkpoint
+    torch_dtype=torch.bfloat16,
+    device_map="cuda",
+)
+tokenizer = AutoTokenizer.from_pretrained("/workspace/models/Qwen3.6-35B-A3B")
+
+# Attach Phase 2 adapter with is_trainable=True
+model = PeftModel.from_pretrained(
+    base_model,
+    "/workspace/checkpoints/daimon_v2_foundation_best",
+    is_trainable=True,  # CRITICAL: without this, all adapter params are frozen
+)
+
+# RED-HAT FIX #2: Verify adapter is trainable (same assert as Phase 2)
+trainable_count = sum(p.numel() for p in model.parameters() if p.requires_grad)
+assert trainable_count > 0, "FATAL: No trainable parameters in Phase 3! Check is_trainable=True."
+expert_lora_count = sum(
+    1 for name, p in model.named_parameters()
+    if "experts" in name and "lora" in name and p.requires_grad
+)
+assert expert_lora_count > 0, "FATAL: No LoRA adapters on expert layers in Phase 3!"
+print(f"Phase 3 trainable params: {trainable_count:,}, expert LoRA params: {expert_lora_count}")
+
+# RED-HAT FIX #2: Weight-delta assert — hash one LoRA tensor, run one step, verify it changed
+import hashlib
+lora_param = next((name, p) for name, p in model.named_parameters()
+                   if "lora" in name and p.requires_grad)
+pre_hash = hashlib.sha256(lora_param[1].data.cpu().numpy().tobytes()).hexdigest()
+
+training_args = SFTConfig(
+    output_dir="/workspace/checkpoints/daimon_v2_calibration",
+    per_device_train_batch_size=2,       # Smaller batch for tiny dataset
+    gradient_accumulation_steps=1,       # Effective batch = 2
+    num_train_epochs=5,                  # More epochs for tiny dataset
+    learning_rate=1e-5,                  # 20x lower than Phase 2
+    lr_scheduler_type="cosine",
+    warmup_ratio=0.1,
+    weight_decay=0.01,
+    bf16=True,
+    max_seq_length=2048,
+    logging_steps=5,
+    save_steps=20,                       # Checkpoint every 20 steps
+    save_total_limit=10,                 # Keep all checkpoints (tiny, <5 GB total)
+    dataloader_num_workers=0,
+    dataset_num_proc=1,
+    gradient_checkpointing=True,
+    gradient_checkpointing_kwargs={"use_reentrant": False},
+    seed=42,
+)
+```
+
+### 6.3 Why This Works
+
+With only 32-40 examples, the risk is memorization rather than generalization. The mitigations:
+
+1. **Low learning rate (1e-5):** 20x lower than Phase 2. The model has already learned the general capability; this stage makes subtle adjustments to the voice distribution.
+2. **The examples are PROCEDURAL:** They teach response patterns (question-before-answer, assess-then-open), not factual content. Procedural patterns generalize better than factual memorization because they operate on structural templates, not surface forms.
+3. **5 epochs is intentional:** Each example is seen 5 times. With LoRA r=32 and DoRA, the adapter capacity is sufficient to learn procedural patterns without memorizing specific words. The cosine LR decay means later epochs make smaller adjustments.
+4. **Recency effect:** This is the last training the model sees. The Thomas voice patterns are the freshest in the gradient signal, which means they have the strongest influence on generation.
+
+### 6.4 Phase 3 Gate
+
+- [ ] Training completed without divergence
+- [ ] Generate the 5 held-out Thomas voice evaluation prompts (not in training data)
+- [ ] Manual review: does it sound like Thomas? Apply the daimon_voice_analysis.md criteria:
+  - Question-before-answer pattern present?
+  - Metaphor usage natural, not forced?
+  - Constructive challenge without preachiness?
+  - Appropriate register (professional, not casual)?
+  - Direct assessment ("The issue is..." "My read is...")?
+- [ ] Compare against Phase 2 output on same prompts -- Phase 3 should be noticeably more "Thomas" without losing capability
+- [ ] Select best checkpoint (likely epoch 3 or 4 based on typical LoRA convergence)
+
+---
+
+## 7. Phase 4: Post-Training {#7-phase-4-post-training}
+
+**Cost:** ~$2-4 (1-2 hours) <!-- RED-HAT FIX #14: Honest after trimming eval battery and cutting LASER from critical path -->
+**Purpose:** Two improvements: OGPSA personality capture and eval battery. LASER moved to optional offline experiment (see 7.2). <!-- RED-HAT FIX #1: LASER cut from critical path per C1 review -->
+
+### 7.1 Merge LoRA into Base Weights
+
+Before post-training, merge the LoRA adapter into the base model. This is required because (a) vLLM cannot load MoE LoRA adapters, and (b) LASER operates on the merged weights.
+
+```python
+# Merge LoRA -> base weights
+model = model.merge_and_unload()
+
+# Save merged model in bf16
+model.save_pretrained(
+    "/workspace/models/daimon_v2_merged",
+    safe_serialization=True,
+)
+tokenizer.save_pretrained("/workspace/models/daimon_v2_merged")
+
+# Verify merged model generates correctly
+inputs = tokenizer("What problem does the dashboard solve?", return_tensors="pt").to("cuda")
+with torch.no_grad():
+    output = model.generate(**inputs, max_new_tokens=200, temperature=0.7)
+print(tokenizer.decode(output[0]))
+```
+
+### 7.2 LASER (Layer-Selective Rank Reduction) — OPTIONAL POST-TRAINING EXPERIMENT
+
+<!-- RED-HAT FIX #1: LASER moved from critical path to optional experiment per C1 review.
+     THREE bugs in the original:
+     1. SVD truncation was INVERTED — zeroed the LARGEST singular values (principal components)
+        instead of the tail. torch.linalg.svd returns values in DESCENDING order.
+        The paper's "higher-order components" means components associated with SMALL singular
+        values, not the highest-magnitude ones.
+     2. Blanket application to ALL 20,480 matrices contradicts LASER's name (LAyer-SElective
+        Rank reduction). The paper's gains come from searching for the specific (layer, matrix)
+        where reduction helps; most choices hurt.
+     3. "+20-30pp" was misquoted — those gains are on narrow factual-recall evals at the single
+        best layer on older models (GPT-J). Realistic upside on a 2026 instruct MoE: 0 to +1-2pp.
+     
+     LASER is NOT free: 20,480 CPU SVDs of 2048x512 fp32 matrices takes 1-3 hours of pod time.
+     The rollback plan already calls it "a free bonus, not a requirement" — it isn't free and
+     as originally written it was negative-value (would have destroyed the trained model).
+     
+     RECOMMENDED: Run LASER locally on Margaret after downloading merged weights ($0 cost).
+     Apply to ONE (layer, matrix) at a time, eval after each, keep only if it helps. -->
+
+**Paper:** arXiv:2312.13558
+**Code:** https://github.com/pratyushasharma/laser
+**Status:** Optional post-budget experiment. Do NOT run on the cloud instance during the primary training run.
+**Realistic upside:** 0 to +1-2pp on narrow factual-recall benchmarks at the single best (layer, matrix). Not a broad reasoning improvement.
+
+**WARNING:** The original code in this spec was inverted and would have destroyed the trained model. The corrected version below keeps the TOP singular values and truncates the TAIL (small singular values = high-order noise).
+
+```python
+# CORRECTED LASER — run locally on Margaret, not on cloud pod
+# RED-HAT FIX #1: Fixed SVD direction. S_modified[n_keep:] = 0 keeps top, drops tail.
+
+import torch
+
+def laser_reduce(weight_matrix, keep_fraction=0.95):
+    """Keep top fraction of singular values, zero out the tail (noise).
+    
+    LASER's insight: small singular values of MLP weight matrices often
+    encode noise. Removing them can improve factual recall.
+    
+    NOTE: torch.linalg.svd returns singular values in DESCENDING order.
+    We keep the first n_keep values (largest) and zero the rest (smallest).
+    """
+    original_dtype = weight_matrix.dtype
+    W = weight_matrix.float()  # SVD requires float32
+    U, S, Vh = torch.linalg.svd(W, full_matrices=False)
+    n_keep = max(1, int(len(S) * keep_fraction))
+    S_modified = S.clone()
+    S_modified[n_keep:] = 0.0  # Zero out TAIL (small values = noise), keep TOP
+    W_modified = U @ torch.diag(S_modified) @ Vh
+    return W_modified.to(original_dtype)
+
+# DO NOT apply blanket to all 20,480 matrices.
+# Apply to ONE (layer, matrix) at a time, eval after each.
+# Example: test on layer 20, gate_proj only:
+#   new_weight = laser_reduce(expert.gate_proj.weight.data, keep_fraction=0.95)
+#   Run eval. If improved, keep. If not, revert.
+```
+
+**LASER protocol (if attempting):**
+1. Download merged weights to Margaret (local, $0)
+2. For each candidate (layer_idx, proj_name) pair, in a systematic sweep:
+   a. Apply `laser_reduce` with `keep_fraction=0.95`
+   b. Run a quick eval subset (100-200 examples)
+   c. Keep only if eval improves over baseline
+3. Try `keep_fraction` values: 0.93, 0.95, 0.97
+4. Apply ONLY to expert FFN (gate_proj, up_proj). Do NOT apply to attention or down_proj.
+
+### 7.3 OGPSA Personality Capture
+
+Per ogpsa_persona_validation_v3.1.md, OGPSA extracts the personality subspace from the model's activations on persona-loaded text.
+
+```python
+# OGPSA Phase 1: Extract persona subspace
+# Use the Thomas voice demonstrations as persona-loaded prompts
+# Plus 10 general QA examples as contrast
+
+# 1. Run persona-loaded examples through the model
+# 2. Capture residual stream activations at all layers
+# 3. SVD on the persona-loaded activation matrix
+# 4. Top 16 components = persona subspace
+# 5. Save as ogpsa_daimon_v2.json
+
+# The OGPSA capture script is at ogpsa_red_team_code.md
+# Adapt for Qwen3.6 architecture (40 layers, not 48)
+```
+
+Output: `ogpsa_daimon_v2.json` -- the personality geometry for drift monitoring.
+
+### 7.4 Evaluation Battery
+
+Run on the merged model (before any optional LASER experiments):
+
+<!-- RED-HAT FIX #7: Removed truthfulqa_mc2 from eval battery per H8 review.
+     TruthfulQA is in the training data (upsampled 15x). Including it in eval
+     measures memorization, not anti-sycophancy generalization. Rely on the
+     custom 10-scenario anti-sycophancy eval instead (well-designed, not contaminated). -->
+
+<!-- RED-HAT FIX #14: Trimmed eval battery per H10 review.
+     Removed arc_easy (redundant with arc_challenge). Increased batch_size to 16.
+     Added --limit 1500 for forgetting detection (5% regression detector, not leaderboard).
+     Single pass only (LASER moved to optional offline experiment). -->
+
+```bash
+# Using lm-evaluation-harness (install in separate venv per H5 review)
+# python -m venv /workspace/lm_eval_venv && source /workspace/lm_eval_venv/bin/activate
+# pip install lm_eval[hf]
+
+# Baseline capabilities (forgetting detection)
+lm_eval --model hf \
+    --model_args pretrained=/workspace/models/daimon_v2_merged,dtype=bfloat16 \
+    --tasks mmlu,gsm8k,hellaswag,arc_challenge,winogrande \
+    --batch_size 16 \
+    --limit 1500 \
+    --output_path /workspace/eval_results/daimon_v2/
+
+# Compare against base model eval (run on unmodified Qwen3.6-35B-A3B for comparison)
+```
+
+**Custom Daimon-specific evals:**
+
+| Eval | Method | Pass Criteria |
+|---|---|---|
+| **Voice attribution** | 5 evaluators, 20 blind response pairs (Daimon vs generic Qwen3.6) | >80% correct attribution to "Thomas-like" |
+| **Daemon function** | 10 scenarios where something is wrong. Does Daimon flag it? | >7/10 flags the concern |
+| **Anti-sycophancy** | 10 scenarios with bad ideas presented as good ones. Does Daimon push back? | >7/10 pushes back |
+| **Anti-nag** | 10 scenarios where everything is fine. Does Daimon stay quiet? | >8/10 does not volunteer unnecessary concerns |
+| **Tool use** | 10 multi-step agentic tasks with function calling | >8/10 correct tool selection and sequencing |
+| **Register calibration** | 5 prompts across different contexts (client, internal, public) | Appropriate register shift in >4/5 |
+
+### 7.5 Phase 4 Gate
+
+- [ ] LoRA merged successfully (model generates coherent text)
+- [ ] LASER: OPTIONAL — moved to post-budget local experiment on Margaret <!-- RED-HAT FIX #1: Per C1 review -->
+- [ ] OGPSA personality subspace captured (ogpsa_daimon_v2.json)
+- [ ] Eval battery results saved (without truthfulqa_mc2 — contaminated) <!-- RED-HAT FIX #7: Per H8 review -->
+- [ ] MMLU within 5% of base model (no catastrophic forgetting)
+- [ ] Custom Daimon evals pass criteria
+- [ ] All artifacts synced off-pod to persistent storage <!-- RED-HAT FIX #8: Per H9 review -->
+
+---
+
+## 8. Phase 5: Export and Validation {#8-phase-5-export-and-validation}
+
+**Cost:** ~$2 (1 hour)
+**Purpose:** Convert to serving format and final validation
+
+### 8.1 Export to GGUF
+
+```bash
+# Clone llama.cpp if not present
+git clone https://github.com/ggerganov/llama.cpp /workspace/llama.cpp
+cd /workspace/llama.cpp && make -j$(nproc)
+
+# Convert to GGUF (bf16 first, then quantize)
+# RED-HAT FIX #1: Export from merged model, not LASER'd model (LASER is optional/offline now)
+python convert_hf_to_gguf.py /workspace/models/daimon_v2_merged \
+    --outfile /workspace/models/daimon_v2.gguf \
+    --outtype bf16
+
+# Quantize to Q4_K_M for serving on Margaret (Mac Studio)
+./llama-quantize /workspace/models/daimon_v2.gguf \
+    /workspace/models/daimon_v2_Q4_K_M.gguf Q4_K_M
+
+# Also export Q5_K_M for higher-quality serving if Margaret has headroom
+./llama-quantize /workspace/models/daimon_v2.gguf \
+    /workspace/models/daimon_v2_Q5_K_M.gguf Q5_K_M
+```
+
+### 8.2 Export to GPTQ/AWQ (Optional, for GPU serving)
+
+```bash
+# If serving on GPU (e.g., via vLLM), export GPTQ
+pip install auto-gptq
+python -c "
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
+
+quantize_config = BaseQuantizeConfig(bits=4, group_size=128, desc_act=False)
+model = AutoGPTQForCausalLM.from_pretrained(
+    '/workspace/models/daimon_v2_merged',  # RED-HAT FIX #1: Use merged model, not LASER'd
+    quantize_config=quantize_config,
+)
+model.quantize()  # Uses calibration data
+model.save_quantized('/workspace/models/daimon_v2_gptq')
+"
+```
+
+### 8.3 Download Artifacts
+
+Before terminating the cloud instance, download all critical artifacts:
+
+```bash
+# Priority 1: The trained model (pick your serving format)
+# GGUF Q4_K_M: ~18 GB
+# GGUF Q5_K_M: ~22 GB
+# bf16 merged: ~70 GB (only if you have bandwidth/storage)
+
+# Priority 2: LoRA checkpoints (for potential continued training)
+# Foundation best: ~500 MB
+# Calibration best: ~500 MB
+
+# Priority 3: Metadata
+# hot_experts.json
+# ogpsa_daimon_v2.json
+# eval_results/ directory
+# training logs
+```
+
+---
+
+## 9. Budget Allocation {#9-budget-allocation}
+
+### Primary Run Budget
+
+<!-- RED-HAT FIX #12: Corrected budget to realistic raw-PEFT throughput per C4 review.
+     Original assumed Unsloth kernel speeds ($20-28 for Phase 2) but shipped code uses
+     raw PEFT + Transformers. Throughput probe (Section 5.5b) gates the actual cost. -->
+
+| Phase | Estimated Time | Cost | Running Total |
+|---|---|---|---|
+| 0: Preflight (incl. backward-pass smoke test) | 30-60 min | $1.50-2.50 | $1.50-2.50 |
+| 1: Expert profiling | 30 min | $1.00 | $2.50-3.50 |
+| 2: Foundation training (gated by throughput probe) | 11-16 hrs | $22-32 | $24.50-35.50 |
+| 3: Thomas voice calibration | 30 min | $1.00 | $25.50-36.50 |
+| 4: Post-training (OGPSA + eval; LASER cut from critical path) | 1-2 hrs | $2-4 | $27.50-40.50 |
+| 5: Export and download | 1 hr | $2-3 | $29.50-43.50 |
+| **Total primary run** | | | **$30-45** |
+
+### Reserve Budget
+
+| Use | Estimated Cost |
+|---|---|
+| **Available reserve** | **$52-67** |
+| Hyperparameter sweep run (if voice quality insufficient) | $22-32 |
+| LR sweep (3 runs at 1e-4, 2e-4, 5e-4) on 10K subset | $6-9 |
+| Rank sweep (r=16 vs r=32 vs r=64) on 10K subset | $6-9 |
+| LASER sweep (offline on Margaret, $0) | $0 |
+| Full re-run with tuned hyperparameters | $22-32 |
+| Emergency buffer | ~$5-15 |
+
+### Cost Optimization Tips
+
+1. **Use spot/interruptible instances** if the provider offers them. Save checkpoints every 2000 steps to survive interruptions.
+2. **Download the model to persistent storage** before the first run. Model download is ~70 GB and takes 15-30 minutes depending on bandwidth. Do not pay H200 rates for downloading.
+3. **Kill the instance between runs** if doing multi-day work. $2/hr idle is $48/day wasted.
+4. **Hyperparameter sweeps on subsets first.** A 10K-sample sweep costs ~$1-2 per run. Do not spend $25 on a full run with untested hyperparameters.
+
+---
+
+## 10. Rollback Plan {#10-rollback-plan}
+
+### Per-Phase Rollback
+
+| Phase | If it fails... | Rollback action | Cost to recover |
+|---|---|---|---|
+| 0: Preflight | Hardware/software mismatch | Fix environment or change provider. No cost wasted (no training started). | $0 |
+| 1: Expert profiling | Profiling script errors | Debug and re-run. Costs ~$0.50. Or fall back to uniform LoRA (all experts) as Tier 1 baseline. | $0.50 |
+| 2: Foundation training | Divergence at step N | Roll back to checkpoint at step N-2000. Adjust LR (halve it) and resume from that checkpoint. Resume costs proportional to remaining steps only. | Variable |
+| 2: Foundation training | OOM/SIGKILL | Reduce batch_size to 2 (effective batch = 8). Or reduce max_seq_length to 1024. Re-run from last checkpoint. | Variable |
+| 2: Foundation training | Poor quality at completion | Try: (a) different LR, (b) longer training (2 epochs), (c) different data mix, (d) fall back to ESFT instead of HELLoRA. Each iteration is $22-32. | $22-32 | <!-- RED-HAT FIX #12: Corrected from $20-28 -->
+| 3: Thomas calibration | Voice too weak | Increase epochs to 10, or increase LR to 5e-5. Re-run from Phase 2 best checkpoint. | $1 |
+| 3: Thomas calibration | Voice too strong (memorized) | Decrease epochs to 3, or decrease LR to 5e-6. Re-run from Phase 2 best checkpoint. | $1 |
+| 4: LASER | Quality degradation | LASER is optional and offline (run on Margaret). Skip entirely if no improvement measured per-layer. | $0 | <!-- RED-HAT FIX #1 -->
+| 4: OGPSA | Capture fails | Re-run capture. If architectural mismatch, adapt the OGPSA script for 40-layer Qwen3.6 (v3.1 was written for a 48-layer model). | $0.50 |
+| 5: Export | GGUF conversion fails | Check llama.cpp supports Qwen3.6 architecture. May need to update llama.cpp to latest. | $0.50 |
+
+### Complete Pipeline Rollback
+
+If the entire pipeline produces unsatisfactory results after the primary run:
+
+1. **Analyze eval results** to identify which capability is deficient (voice, tool use, anti-sycophancy, reasoning).
+2. **Targeted fix options:**
+   - Voice weak: More Thomas demonstrations + higher LR in Phase 3
+   - Tool use weak: Upsample xlam data to 2x in Phase 2 mix
+   - Anti-sycophancy weak: Upsample TruthfulQA to 25x; consider adding SAA dataset
+   - Reasoning weak: Upsample hermes traces to 5x; consider adding GSM8K training data
+3. **Full re-run** with adjusted data mix: ~$30-45 from reserve budget. <!-- RED-HAT FIX #12: Corrected from $27-37 -->
+4. **Method pivot** if HELLoRA ceiling is too low: Switch to ESFT (full fine-tune hot experts) or BAdam (block-coordinate). This is a larger change and costs one full run.
+
+### Checkpoint Strategy
+
+Checkpoints are the most important insurance:
+
+- Phase 2 saves every 2000 steps (5 checkpoints retained)
+- Phase 3 saves every 20 steps (all checkpoints retained -- tiny files)
+- The BEST checkpoint from each phase is copied to a separate directory
+- All checkpoints include the full LoRA adapter state + optimizer state (allows resuming training)
+- **Checkpoints synced off-pod after every save** (see Section 5.6b) <!-- RED-HAT FIX #8: Per H9 review -->
+- Download at least the best checkpoints to local storage before terminating the instance
+
+---
+
+## Appendix A: Environment Setup Script {#appendix-a}
+
+```bash
+#!/bin/bash
+# daimon_setup.sh -- Run this FIRST on a fresh H200 instance
+set -euo pipefail
+
+echo "=== Daimon Training Pipeline: Environment Setup ==="
+
+# 1. System checks
+echo "--- Hardware checks ---"
+nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
+CONTAINER_RAM=$(cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes 2>/dev/null)
+echo "Container RAM: $(echo "$CONTAINER_RAM / 1024 / 1024 / 1024" | bc) GB"
+
+# 2. Set environment variables BEFORE any Python imports
+export UNSLOTH_COMPILE_DISABLE=1
+export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
+export TOKENIZERS_PARALLELISM=false
+export UNSLOTH_DISABLE_FAST_GENERATION=1
+
+# Persist for all subsequent shells
+cat >> ~/.bashrc << 'ENVEOF'
+export UNSLOTH_COMPILE_DISABLE=1
+export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
+export TOKENIZERS_PARALLELISM=false
+export UNSLOTH_DISABLE_FAST_GENERATION=1
+ENVEOF
+
+# 3. Install/update packages
+# RED-HAT FIX #9: Pin exact versions and install torch FIRST per H5 review.
+# Original had version floors (>=) with torch installed LAST, causing resolution chaos.
+# Install order matters: torch first (pinned to cu124), then core ML stack, unsloth --no-deps last.
+# lm_eval in separate venv to avoid dependency conflicts.
+
+# Ensure conda is on PATH (some cloud images need this)
+export PATH="/opt/conda/bin:$PATH"
+
+# Set HF cache to /workspace to avoid filling root partition (RED-HAT FIX #5, C5)
+export HF_HOME=/workspace/.hf
+export HF_DATASETS_CACHE=/workspace/.hf/datasets
+mkdir -p "$HF_HOME" "$HF_DATASETS_CACHE"
+cat >> ~/.bashrc << 'HFEOF'
+export HF_HOME=/workspace/.hf
+export HF_DATASETS_CACHE=/workspace/.hf/datasets
+HFEOF
+
+pip install --upgrade pip
+
+# Step 1: Install torch FIRST with CUDA 12.4 (pinned version)
+pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu124
+
+# Step 2: Verify torch installed correctly BEFORE proceeding
+python -c "import torch; assert torch.cuda.is_available(), 'CUDA not available after torch install'; print(f'torch {torch.__version__} OK')"
+
+# Step 3: Install core ML stack (pinned versions, no torch re-resolution)
+pip install transformers==5.0.2
+pip install peft==0.15.2
+pip install trl==0.18.1
+pip install datasets==3.6.0
+pip install accelerate==1.7.0
+pip install bitsandbytes==0.46.0  # Required by unsloth even if we don't use 4-bit
+pip install sentencepiece protobuf
+
+# Step 4: Install unsloth LAST with --no-deps to avoid overwriting pinned versions
+pip install "unsloth>=0.1.47" --no-deps
+
+# Step 5: Verify no dependency conflicts
+pip check || echo "WARNING: pip check found conflicts — review before proceeding"
+
+# Step 6: Freeze the resolved environment for reproducibility
+pip freeze > /workspace/artifacts/requirements.lock
+echo "Locked environment saved to /workspace/artifacts/requirements.lock"
+
+# Step 7: Install lm_eval in a SEPARATE venv to avoid conflicts (RED-HAT FIX #9, H5)
+python -m venv /workspace/lm_eval_venv
+/workspace/lm_eval_venv/bin/pip install --upgrade pip
+/workspace/lm_eval_venv/bin/pip install "lm_eval[hf]"
+
+# 4. Verify installations
+python -c "
+import os
+os.environ['UNSLOTH_COMPILE_DISABLE'] = '1'
+os.environ['PYTORCH_CUDA_ALLOC_CONF'] = 'expandable_segments:True'
+os.environ['TOKENIZERS_PARALLELISM'] = 'false'
+os.environ['UNSLOTH_DISABLE_FAST_GENERATION'] = '1'
+
+import torch
+print(f'PyTorch {torch.__version__}, CUDA available: {torch.cuda.is_available()}')
+if torch.cuda.is_available():
+    print(f'GPU: {torch.cuda.get_device_name(0)}')
+    print(f'VRAM: {torch.cuda.get_device_properties(0).total_memory / 1e9:.1f} GB')
+
+import transformers; print(f'Transformers {transformers.__version__}')
+import peft; print(f'PEFT {peft.__version__}')
+import unsloth; print(f'Unsloth {unsloth.__version__}')
+print('All imports successful.')
+"
+
+# 5. Create workspace directories
+mkdir -p /workspace/models
+mkdir -p /workspace/data
+mkdir -p /workspace/checkpoints
+mkdir -p /workspace/eval_results
+mkdir -p /workspace/artifacts
+
+echo "=== Setup complete ==="
+```
+
+---
+
+## Appendix B: Expert Profiling Script {#appendix-b}
+
+```python
+#!/usr/bin/env python3
+"""
+daimon_expert_profiling.py
+Phase 1: Profile which experts activate for Daimon's training data.
+Uses ESFT-style profiling with MAN (Mean Activation Norm) scoring.
+
+Reference: ESFT (arXiv:2407.01906), Unified Expert Scoring (arXiv:2606.15716)
+"""
+
+import os
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"
+
+import json
+import torch
+import numpy as np
+from collections import defaultdict
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_PATH = "/workspace/models/Qwen3.6-35B-A3B"
+OUTPUT_PATH = "/workspace/artifacts/hot_experts.json"
+NUM_LAYERS = 40
+NUM_EXPERTS = 256
+TOP_K = 8  # Qwen3.6 routes to top-8 experts
+
+# Representative samples from each data source
+# Adjust paths to match your data layout
+SAMPLE_SOURCES = {
+    "sonnet_voice": ("/workspace/data/sonnet_voice_distill/", 8),
+    "prosocial": ("/workspace/data/prosocial_dialogue/", 8),
+    "tool_use": ("/workspace/data/xlam_function_calling/", 8),
+    "reasoning": ("/workspace/data/hermes_agent_traces/", 4),
+    "thomas_voice": ("/workspace/data/thomas_voice_demos/daimon_voice_demonstrations.jsonl", 4),
+}
+
+def load_samples():
+    """Load representative samples from each data source."""
+    samples = []
+    for source_name, (path, count) in SAMPLE_SOURCES.items():
+        # Load first `count` examples from each source
+        # Adapt this loader to match your data format
+        if path.endswith(".jsonl"):
+            with open(path) as f:
+                for i, line in enumerate(f):
+                    if i >= count:
+                        break
+                    data = json.loads(line)
+                    # Extract the full conversation as a single string
+                    text = " ".join(m["content"] for m in data["messages"])
+                    samples.append(text)
+        else:
+            # Directory of files -- load from first file found
+            import glob
+            files = sorted(glob.glob(os.path.join(path, "*.jsonl")))
+            if not files:
+                files = sorted(glob.glob(os.path.join(path, "*.json")))
+            if files:
+                loaded = 0
+                for fpath in files:
+                    with open(fpath) as f:
+                        for line in f:
+                            if loaded >= count:
+                                break
+                            data = json.loads(line)
+                            if "messages" in data:
+                                text = " ".join(m["content"] for m in data["messages"])
+                            elif "text" in data:
+                                text = data["text"]
+                            elif "conversations" in data:
+                                text = " ".join(c.get("value", "") for c in data["conversations"])
+                            else:
+                                text = str(data)
+                            samples.append(text)
+                            loaded += 1
+                    if loaded >= count:
+                        break
+    print(f"Loaded {len(samples)} profiling samples")
+    return samples
+
+
+def profile_experts(model, tokenizer, samples):
+    """
+    Profile expert activation patterns using Mean Activation Norm (MAN).
+
+    For each MoE layer, record the activation norm of each expert's output
+    when it is selected by the router. The Mean Activation Norm across all
+    samples gives the expert importance score.
+    """
+    # Storage: layer_idx -> expert_idx -> list of activation norms
+    activation_norms = defaultdict(lambda: defaultdict(list))
+
+    # Register hooks on MoE layers
+    hooks = []
+
+    def make_hook(layer_idx):
+        def hook_fn(module, input, output):
+            # The MoE module's output includes routing information
+            # We need to capture which experts were selected and their output norms
+            # This hook structure depends on the exact Qwen3.6 implementation
+            #
+            # For Qwen3.6, the MoE block routes tokens and returns the weighted sum.
+            # To get per-expert activation norms, we need to hook deeper -- at the
+            # router level and individual expert level.
+            pass
+        return hook_fn
+
+    # Alternative approach: direct router inspection
+    # Hook the router to get expert selection, then measure expert output norms
+    for layer_idx in range(NUM_LAYERS):
+        moe_layer = model.model.layers[layer_idx].mlp
+
+        # Hook the gate (router) to capture routing decisions
+        def make_router_hook(l_idx):
+            def hook_fn(module, input, output):
+                # output is the router logits or routing weights
+                # For Qwen3.6: output shape is [batch*seq_len, num_experts]
+                if isinstance(output, tuple):
+                    router_logits = output[0]
+                else:
+                    router_logits = output
+
+                # Get top-k expert indices
+                topk_vals, topk_indices = torch.topk(router_logits, TOP_K, dim=-1)
+
+                # Record which experts were selected and their gate values
+                for expert_idx in range(NUM_EXPERTS):
+                    mask = (topk_indices == expert_idx).any(dim=-1)
+                    if mask.any():
+                        # Use gate value as proxy for activation norm
+                        # (actual activation norm requires hooking each expert separately)
+                        gate_vals = topk_vals[mask]
+                        mean_gate = gate_vals.mean().item()
+                        activation_norms[l_idx][expert_idx].append(mean_gate)
+
+            return hook_fn
+
+        hook = moe_layer.gate.register_forward_hook(make_router_hook(layer_idx))
+        hooks.append(hook)
+
+    # Run inference on all samples
+    model.eval()
+    with torch.no_grad():
+        for i, text in enumerate(samples):
+            inputs = tokenizer(
+                text, return_tensors="pt", truncation=True, max_length=2048
+            ).to("cuda")
+            _ = model(**inputs)
+            if i % 8 == 0:
+                print(f"Profiled sample {i+1}/{len(samples)}")
+
+    # Remove hooks
+    for hook in hooks:
+        hook.remove()
+
+    return activation_norms
+
+
+def compute_hot_experts(activation_norms, coverage_threshold=0.80):
+    """
+    Compute hot experts per layer using Mean Activation Norm.
+
+    For each layer, sort experts by MAN score and select the minimum set
+    that covers >= coverage_threshold of total activation norm.
+    """
+    hot_experts = {}
+
+    for layer_idx in range(NUM_LAYERS):
+        expert_scores = {}
+        for expert_idx in range(NUM_EXPERTS):
+            norms = activation_norms[layer_idx].get(expert_idx, [])
+            if norms:
+                expert_scores[expert_idx] = np.mean(norms)
+            else:
+                expert_scores[expert_idx] = 0.0
+
+        # Sort by MAN score descending
+        sorted_experts = sorted(expert_scores.items(), key=lambda x: x[1], reverse=True)
+        total_norm = sum(score for _, score in sorted_experts)
+
+        if total_norm == 0:
+            print(f"WARNING: Layer {layer_idx} has zero total activation norm!")
+            hot_experts[layer_idx] = list(range(TOP_K))  # Fallback
+            continue
+
+        # Select experts until coverage threshold is met
+        cumulative = 0.0
+        hot = []
+        for expert_idx, score in sorted_experts:
+            hot.append(expert_idx)
+            cumulative += score
+            if cumulative / total_norm >= coverage_threshold:
+                break
+
+        hot_experts[layer_idx] = sorted(hot)
+        print(f"Layer {layer_idx}: {len(hot)} hot experts "
+              f"({100*len(hot)/NUM_EXPERTS:.1f}% of 256, "
+              f"covering {100*cumulative/total_norm:.1f}% of activation norm)")
+
+    return hot_experts
+
+
+def main():
+    print("=== Phase 1: Expert Profiling ===")
+
+    # Load model for inference only
+    print("Loading model...")
+    tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)
+    model = AutoModelForCausalLM.from_pretrained(
+        MODEL_PATH,
+        torch_dtype=torch.bfloat16,
+        device_map="cuda",
+    )
+
+    # Load profiling samples
+    samples = load_samples()
+
+    # Profile
+    print("Profiling expert activations...")
+    activation_norms = profile_experts(model, tokenizer, samples)
+
+    # Compute hot experts
+    print("\nComputing hot experts (80% coverage threshold)...")
+    hot_experts = compute_hot_experts(activation_norms, coverage_threshold=0.80)
+
+    # Summary statistics
+    counts = [len(v) for v in hot_experts.values()]
+    print(f"\nHot experts per layer: min={min(counts)}, max={max(counts)}, "
+          f"mean={np.mean(counts):.1f}, median={np.median(counts):.1f}")
+
+    total_hot = sum(counts)
+    total_possible = NUM_LAYERS * NUM_EXPERTS
+    print(f"Total hot expert slots: {total_hot}/{total_possible} "
+          f"({100*total_hot/total_possible:.1f}%)")
+
+    # Save
+    # Convert int keys to strings for JSON
+    hot_experts_json = {str(k): v for k, v in hot_experts.items()}
+    with open(OUTPUT_PATH, "w") as f:
+        json.dump(hot_experts_json, f, indent=2)
+    print(f"\nSaved to {OUTPUT_PATH}")
+
+    # Sanity checks
+    for layer_idx, experts in hot_experts.items():
+        if len(experts) < 5:
+            print(f"WARNING: Layer {layer_idx} has only {len(experts)} hot experts. "
+                  "Consider lowering coverage threshold.")
+        if len(experts) > 100:
+            print(f"WARNING: Layer {layer_idx} has {len(experts)} hot experts. "
+                  "Routing may be too diffuse. Check for profiling errors.")
+
+    del model
+    torch.cuda.empty_cache()
+    print("\n=== Expert profiling complete ===")
+
+
+if __name__ == "__main__":
+    main()
+```
+
+**Important notes on the profiling script:**
+
+1. The router hook structure assumes Qwen3.6's MoE implementation exposes router logits through the gate module. The exact module hierarchy may differ -- inspect the model's `named_modules()` output to find the correct hook points.
+2. Using gate values as a proxy for activation norms is an approximation. For exact MAN scoring per arXiv:2606.15716, hook each expert's output layer and measure the L2 norm of the output tensor. This requires more memory but gives more accurate scores.
+3. The 80% coverage threshold is the starting point. If it selects >50 experts per layer, raise to 85%. If it selects <15, lower to 75%.
+
+---
+
+## Appendix C: Training Script {#appendix-c}
+
+```python
+#!/usr/bin/env python3
+"""
+daimon_train.py
+Phase 2 + Phase 3: Foundation training + Thomas voice calibration.
+
+Usage:
+    python daimon_train.py --phase foundation
+    python daimon_train.py --phase calibration --checkpoint /path/to/best
+"""
+
+import os
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+os.environ["TOKENIZERS_PARALLELISM"] = "false"
+os.environ["UNSLOTH_DISABLE_FAST_GENERATION"] = "1"
+
+import json
+import argparse
+import torch
+from datasets import load_dataset, interleave_datasets, Dataset
+from peft import LoraConfig, get_peft_model, PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from trl import SFTTrainer, SFTConfig
+
+
+def build_target_modules(hot_experts_path):
+    """Build HELLoRA target module list from expert profiling results."""
+    with open(hot_experts_path) as f:
+        hot_experts = json.load(f)
+
+    target_modules = []
+
+    # Attention projections (all layers)
+    for layer_idx in range(40):
+        for proj in ["q_proj", "k_proj", "v_proj", "o_proj"]:
+            target_modules.append(f"model.layers.{layer_idx}.self_attn.{proj}")
+
+    # Hot expert FFN layers only
+    for layer_idx_str, expert_indices in hot_experts.items():
+        layer_idx = int(layer_idx_str)
+        for expert_idx in expert_indices:
+            for proj in ["gate_proj", "up_proj", "down_proj"]:
+                target_modules.append(
+                    f"model.layers.{layer_idx}.mlp.experts.{expert_idx}.{proj}"
+                )
+
+    return target_modules
+
+
+def build_foundation_dataset(tokenizer):
+    """Build the mixed foundation dataset with appropriate sampling.
+    
+    RED-HAT FIX #3: Returns (train_dataset, eval_dataset) — stratified eval split
+    of ~300 examples held out before interleaving. (Per C3 review)
+    """
+    # Load each dataset source
+    # NOTE: Adapt these loaders to match your actual data format and paths
+
+    # RED-HAT FIX #3: Hold out eval examples BEFORE interleaving.
+    # ~50 from each major source, fewer from small sources, fixed seed.
+    EVAL_COUNTS = {
+        "sonnet": 80, "prosocial": 50, "xlam": 50,
+        "hermes": 30, "code": 30, "truthful": 60
+    }
+
+    datasets_with_weights = []
+    eval_datasets = []
+
+    # 1. Sonnet voice distill (226K, weight 0.44)
+    sonnet_ds = load_dataset("json", data_files="/workspace/data/sonnet_voice_distill/*.jsonl",
+                              split="train").shuffle(seed=42)
+    eval_datasets.append(sonnet_ds.select(range(EVAL_COUNTS["sonnet"])))
+    sonnet_ds = sonnet_ds.select(range(EVAL_COUNTS["sonnet"], len(sonnet_ds)))
+    datasets_with_weights.append((sonnet_ds, 0.44))
+
+    # 2. Prosocial dialogue (165K subsampled to 49.5K, weight 0.10)
+    prosocial_ds = load_dataset("json", data_files="/workspace/data/prosocial_dialogue/*.jsonl",
+                                 split="train").shuffle(seed=42)
+    eval_datasets.append(prosocial_ds.select(range(EVAL_COUNTS["prosocial"])))
+    prosocial_ds = prosocial_ds.select(range(EVAL_COUNTS["prosocial"], len(prosocial_ds)))
+    prosocial_ds = prosocial_ds.select(range(min(49500, len(prosocial_ds))))
+    datasets_with_weights.append((prosocial_ds, 0.10))
+
+    # 3. xlam function calling (60K, weight 0.12)
+    xlam_ds = load_dataset("json", data_files="/workspace/data/xlam_function_calling/*.jsonl",
+                            split="train").shuffle(seed=42)
+    eval_datasets.append(xlam_ds.select(range(EVAL_COUNTS["xlam"])))
+    xlam_ds = xlam_ds.select(range(EVAL_COUNTS["xlam"], len(xlam_ds)))
+    datasets_with_weights.append((xlam_ds, 0.12))
+
+    # 4. Hermes agent traces (7.6K upsampled 3x, weight 0.04)
+    hermes_ds = load_dataset("json", data_files="/workspace/data/hermes_agent_traces/*.jsonl",
+                              split="train").shuffle(seed=42)
+    eval_datasets.append(hermes_ds.select(range(EVAL_COUNTS["hermes"])))
+    hermes_ds = hermes_ds.select(range(EVAL_COUNTS["hermes"], len(hermes_ds)))
+    hermes_upsampled = Dataset.from_dict({
+        k: hermes_ds[k] * 3 for k in hermes_ds.column_names
+    })
+    datasets_with_weights.append((hermes_upsampled, 0.04))
+
+    # 5. Code review feedback (9.5K upsampled 2x, weight 0.04)
+    code_ds = load_dataset("json", data_files="/workspace/data/code_review_feedback/*.jsonl",
+                            split="train").shuffle(seed=42)
+    eval_datasets.append(code_ds.select(range(EVAL_COUNTS["code"])))
+    code_ds = code_ds.select(range(EVAL_COUNTS["code"], len(code_ds)))
+    code_upsampled = Dataset.from_dict({
+        k: code_ds[k] * 2 for k in code_ds.column_names
+    })
+    datasets_with_weights.append((code_upsampled, 0.04))
+
+    # 6. TruthfulQA (817 upsampled 15x, weight 0.02)
+    truthful_ds = load_dataset("json", data_files="/workspace/data/truthfulqa/*.jsonl",
+                                split="train").shuffle(seed=42)
+    eval_datasets.append(truthful_ds.select(range(EVAL_COUNTS["truthful"])))
+    truthful_ds = truthful_ds.select(range(EVAL_COUNTS["truthful"], len(truthful_ds)))
+    truthful_upsampled = Dataset.from_dict({
+        k: truthful_ds[k] * 15 for k in truthful_ds.column_names
+    })
+    datasets_with_weights.append((truthful_upsampled, 0.02))
+
+    # Interleave training data with weights
+    all_datasets = [d for d, w in datasets_with_weights]
+    all_weights = [w for d, w in datasets_with_weights]
+
+    # Normalize weights
+    total_weight = sum(all_weights)
+    probabilities = [w / total_weight for w in all_weights]
+
+    mixed_dataset = interleave_datasets(
+        all_datasets,
+        probabilities=probabilities,
+        seed=42,
+        stopping_strategy="all_exhausted",
+    )
+
+    # RED-HAT FIX #3: Combine eval splits into a single eval dataset
+    from datasets import concatenate_datasets
+    eval_dataset = concatenate_datasets(eval_datasets).shuffle(seed=42)
+
+    print(f"Foundation train dataset: {len(mixed_dataset)} examples")
+    print(f"Foundation eval dataset: {len(eval_dataset)} examples")
+    return mixed_dataset, eval_dataset
+
+
+def train_foundation(model_path, hot_experts_path):
+    """Phase 2: Foundation training."""
+    print("=== Phase 2: Foundation Training ===")
+
+    # Load model
+    tokenizer = AutoTokenizer.from_pretrained(model_path)
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.bfloat16,
+        device_map="cuda",
+    )
+
+    # Apply HELLoRA
+    target_modules = build_target_modules(hot_experts_path)
+    print(f"Target modules: {len(target_modules)}")
+
+    lora_config = LoraConfig(
+        r=32,
+        lora_alpha=64,
+        lora_dropout=0.05,
+        target_modules=target_modules,
+        use_dora=True,
+        bias="none",
+        task_type="CAUSAL_LM",
+    )
+
+    model = get_peft_model(model, lora_config)
+    model.print_trainable_parameters()
+
+    # Verify expert LoRA adapters exist
+    expert_lora_count = sum(
+        1 for name, p in model.named_parameters()
+        if "experts" in name and "lora" in name and p.requires_grad
+    )
+    assert expert_lora_count > 0, "FATAL: No LoRA adapters on expert layers!"
+    print(f"Expert LoRA parameters: {expert_lora_count}")
+
+    # Build dataset
+    # RED-HAT FIX #3: build_foundation_dataset now returns (train_dataset, eval_dataset)
+    # Eval split: ~300 examples stratified across all 6 sources, fixed seed, excluded from training.
+    train_dataset, eval_dataset = build_foundation_dataset(tokenizer)
+
+    # Training config
+    training_args = SFTConfig(
+        output_dir="/workspace/checkpoints/daimon_v2_foundation",
+        per_device_train_batch_size=4,
+        gradient_accumulation_steps=4,
+        num_train_epochs=1,
+        learning_rate=2e-4,
+        lr_scheduler_type="cosine",
+        warmup_ratio=0.03,
+        weight_decay=0.01,
+        bf16=True,
+        max_seq_length=2048,
+        logging_steps=50,
+        save_steps=2000,
+        save_total_limit=5,
+        dataloader_num_workers=0,
+        dataset_num_proc=1,
+        gradient_checkpointing=True,
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        eval_strategy="steps",
+        eval_steps=2000,
+        load_best_model_at_end=True,             # RED-HAT FIX #3: Select best checkpoint by eval_loss
+        metric_for_best_model="eval_loss",        # RED-HAT FIX #3
+        max_steps=30000,                          # RED-HAT FIX #4: Cost circuit breaker (adjusted by throughput probe)
+        report_to="none",
+        seed=42,
+    )
+
+    # Train
+    trainer = SFTTrainer(
+        model=model,
+        args=training_args,
+        train_dataset=train_dataset,              # RED-HAT FIX #3: Was `dataset`, now split
+        eval_dataset=eval_dataset,                # RED-HAT FIX #3: Added — prevents eval crash
+        processing_class=tokenizer,
+    )
+
+    trainer.train()
+
+    # Save best checkpoint
+    model.save_pretrained("/workspace/checkpoints/daimon_v2_foundation_best")
+    tokenizer.save_pretrained("/workspace/checkpoints/daimon_v2_foundation_best")
+    print("=== Foundation training complete ===")
+
+
+def train_calibration(checkpoint_path, base_model_path="/workspace/models/Qwen3.6-35B-A3B"):
+    """Phase 3: Thomas voice calibration.
+    
+    RED-HAT FIX #2: Phase 2 saves adapter-only (adapter_config.json + adapter_model.safetensors),
+    NOT a full model. Must load base model first, then attach adapter with is_trainable=True.
+    Original code used AutoModelForCausalLM.from_pretrained on the adapter dir, which either
+    crashes ("no config.json") or loads adapter for inference-only (requires_grad=False),
+    causing silent no-op training — the exact mlx-lm #571 failure class. (Per C2 review)
+    """
+    print("=== Phase 3: Thomas Voice Calibration ===")
+
+    # RED-HAT FIX #2: Load BASE model first, then attach Phase 2 adapter
+    tokenizer = AutoTokenizer.from_pretrained(base_model_path)
+    base_model = AutoModelForCausalLM.from_pretrained(
+        base_model_path,  # Base model path, NOT the checkpoint
+        torch_dtype=torch.bfloat16,
+        device_map="cuda",
+    )
+    
+    # Attach Phase 2 adapter with is_trainable=True
+    model = PeftModel.from_pretrained(
+        base_model,
+        checkpoint_path,     # Phase 2 adapter checkpoint
+        is_trainable=True,   # CRITICAL: without this, all adapter params are frozen
+    )
+    
+    # RED-HAT FIX #2: Verify adapter is actually trainable (catches silent no-op)
+    trainable_count = sum(p.numel() for p in model.parameters() if p.requires_grad)
+    assert trainable_count > 0, "FATAL: No trainable parameters in Phase 3! Check is_trainable=True."
+    expert_lora_count = sum(
+        1 for name, p in model.named_parameters()
+        if "experts" in name and "lora" in name and p.requires_grad
+    )
+    assert expert_lora_count > 0, "FATAL: No LoRA adapters on expert layers in Phase 3!"
+    print(f"Phase 3 trainable params: {trainable_count:,}, expert LoRA params: {expert_lora_count}")
+    
+    # RED-HAT FIX #2: Weight-delta assert — verify training actually changes weights
+    import hashlib
+    lora_param = next((name, p) for name, p in model.named_parameters()
+                       if "lora" in name and p.requires_grad)
+    pre_hash = hashlib.sha256(lora_param[1].data.cpu().numpy().tobytes()).hexdigest()
+
+    # Load Thomas voice demonstrations
+    demos = []
+    with open("/workspace/data/thomas_voice_demos/daimon_voice_demonstrations.jsonl") as f:
+        for line in f:
+            demos.append(json.loads(line))
+    dataset = Dataset.from_list(demos)
+    print(f"Thomas voice demos: {len(dataset)} examples")
+
+    training_args = SFTConfig(
+        output_dir="/workspace/checkpoints/daimon_v2_calibration",
+        per_device_train_batch_size=2,
+        gradient_accumulation_steps=1,
+        num_train_epochs=5,
+        learning_rate=1e-5,
+        lr_scheduler_type="cosine",
+        warmup_ratio=0.1,
+        weight_decay=0.01,
+        bf16=True,
+        max_seq_length=2048,
+        logging_steps=5,
+        save_steps=20,
+        save_total_limit=10,
+        dataloader_num_workers=0,
+        dataset_num_proc=1,
+        gradient_checkpointing=True,
+        gradient_checkpointing_kwargs={"use_reentrant": False},
+        seed=42,
+    )
+
+    trainer = SFTTrainer(
+        model=model,
+        args=training_args,
+        train_dataset=dataset,
+        processing_class=tokenizer,
+    )
+
+    trainer.train()
+    
+    # RED-HAT FIX #2: Verify weight actually changed after training
+    post_hash = hashlib.sha256(lora_param[1].data.cpu().numpy().tobytes()).hexdigest()
+    assert pre_hash != post_hash, (
+        f"FATAL: LoRA weight {lora_param[0]} unchanged after 5 epochs of training! "
+        "Phase 3 was a no-op. Check adapter loading."
+    )
+    print(f"Weight-delta verified: {lora_param[0]} changed")
+
+    model.save_pretrained("/workspace/checkpoints/daimon_v2_calibration_best")
+    tokenizer.save_pretrained("/workspace/checkpoints/daimon_v2_calibration_best")
+    print("=== Thomas voice calibration complete ===")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--phase", choices=["foundation", "calibration"], required=True)
+    parser.add_argument("--model", default="/workspace/models/Qwen3.6-35B-A3B")
+    parser.add_argument("--checkpoint", default=None, help="Checkpoint for calibration phase")
+    parser.add_argument("--hot-experts", default="/workspace/artifacts/hot_experts.json")
+    args = parser.parse_args()
+
+    if args.phase == "foundation":
+        train_foundation(args.model, args.hot_experts)
+    elif args.phase == "calibration":
+        cp = args.checkpoint or "/workspace/checkpoints/daimon_v2_foundation_best"
+        train_calibration(cp)
+```
+
+**Critical implementation notes:**
+
+1. **Unsloth integration:** The script above uses raw PEFT + Transformers for clarity. In production, you may want to use Unsloth's `FastLanguageModel` for the 12x MoE kernel speedup. If using Unsloth, verify that `FastLanguageModel.get_peft_model()` accepts explicit module name lists (not just module type strings). If it does not, use the PEFT-direct approach shown here and accept the throughput penalty, or patch Unsloth's module targeting.
+
+2. **Data format compatibility:** The `SFTTrainer` expects data in a specific format depending on the `dataset_text_field` or `formatting_func` parameter. If your data uses `{"messages": [...]}` format, you may need to set `dataset_text_field=None` and let TRL auto-detect the chat format, or provide a custom `formatting_func` that applies the Qwen3.6 chat template.
+
+3. **Gradient checkpointing:** `use_reentrant=False` is required for compatibility with PEFT LoRA on MoE. The reentrant variant can cause incorrect gradients when LoRA interacts with MoE routing.
+
+4. **Resuming from interruption:** If training is interrupted (spot instance preemption, SIGKILL, etc.), resume from the last checkpoint:
+   ```bash
+   python daimon_train.py --phase foundation --checkpoint /workspace/checkpoints/daimon_v2_foundation/checkpoint-XXXX
+   ```
+   Add `resume_from_checkpoint=True` to the SFTConfig, or pass the checkpoint path to `trainer.train(resume_from_checkpoint=...)`.
+
+---
+
+## Appendix D: LASER Post-Training Script (OPTIONAL — Run Locally on Margaret) {#appendix-d}
+
+<!-- RED-HAT FIX #1: LASER moved to optional offline experiment per C1 review.
+     Original code was INVERTED — zeroed the LARGEST singular values (principal components)
+     instead of the tail. This would have destroyed the trained model's quality.
+     
+     Additionally, blanket application to all 20,480 matrices contradicts LASER's
+     LAyer-SElective protocol. The corrected script below:
+     1. Fixes SVD direction: keeps top singular values, zeros the tail
+     2. Applies per-(layer, matrix) with eval between each application
+     3. Designed for local execution on Margaret ($0), not the cloud pod -->
+
+```python
+#!/usr/bin/env python3
+"""
+daimon_laser.py
+OPTIONAL post-budget experiment: LASER (Layer-Selective Rank Reduction) on merged model.
+Run LOCALLY on Margaret, not on cloud pod.
+
+Reference: arXiv:2312.13558
+
+RED-HAT FIX #1 (C1 review): SVD direction corrected.
+Original code zeroed S[:n_remove] which removes the LARGEST singular values
+(principal components). torch.linalg.svd returns values in DESCENDING order.
+Corrected: S[n_keep:] = 0 keeps the top (largest) and zeros the tail (smallest = noise).
+"""
+
+import os
+os.environ["UNSLOTH_COMPILE_DISABLE"] = "1"
+os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"
+
+import torch
+import argparse
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+NUM_LAYERS = 40
+NUM_EXPERTS = 256
+
+
+def laser_reduce(weight_matrix, keep_fraction=0.95):
+    """
+    Keep top fraction of singular values, zero out the tail (noise).
+
+    LASER's key insight: the SMALL singular values of MLP weight matrices
+    often encode noise or spurious correlations. Removing them (keeping
+    only the top components) can improve factual recall on specific tasks.
+
+    IMPORTANT: torch.linalg.svd returns singular values in DESCENDING order.
+    S[0] is the LARGEST. We keep S[:n_keep] (largest) and zero S[n_keep:] (smallest).
+
+    The ORIGINAL code in this spec had this INVERTED (S[:n_remove] = 0), which
+    would have destroyed the principal components of every weight matrix.
+
+    Args:
+        weight_matrix: 2D tensor [out_features, in_features]
+        keep_fraction: fraction of top singular values to KEEP (default: 0.95 = drop bottom 5%)
+    Returns:
+        Modified weight matrix with tail singular values removed
+    """
+    original_dtype = weight_matrix.dtype
+    W = weight_matrix.float()  # SVD requires float32
+
+    U, S, Vh = torch.linalg.svd(W, full_matrices=False)
+    n_keep = max(1, int(len(S) * keep_fraction))
+
+    S_modified = S.clone()
+    S_modified[n_keep:] = 0.0  # Zero out TAIL (small values = noise), KEEP TOP
+
+    W_modified = U @ torch.diag(S_modified) @ Vh
+    return W_modified.to(original_dtype)
+
+
+def apply_laser_selective(model_path, output_path, keep_fraction=0.95,
+                          target_layer=None, target_proj="gate_proj"):
+    """
+    Apply LASER to a SINGLE (layer, projection) pair.
+
+    DO NOT apply blanket to all 20,480 matrices — that contradicts LASER's
+    LAyer-SElective protocol. Apply one at a time, eval after each.
+    """
+    print(f"=== LASER: keep_fraction={keep_fraction}, layer={target_layer}, proj={target_proj} ===")
+
+    tokenizer = AutoTokenizer.from_pretrained(model_path)
+    model = AutoModelForCausalLM.from_pretrained(
+        model_path,
+        torch_dtype=torch.bfloat16,
+        device_map="cpu",  # LASER on CPU — no GPU needed
+    )
+
+    if target_layer is not None:
+        layers_to_process = [target_layer]
+    else:
+        layers_to_process = range(NUM_LAYERS)
+
+    total_modified = 0
+    for layer_idx in layers_to_process:
+        moe_block = model.model.layers[layer_idx].mlp
+
+        for expert_idx in range(NUM_EXPERTS):
+            expert = moe_block.experts[expert_idx]
+            weight = getattr(expert, target_proj).weight.data
+            new_weight = laser_reduce(weight, keep_fraction)
+            getattr(expert, target_proj).weight.data = new_weight
+            total_modified += 1
+
+        print(f"  Layer {layer_idx}: modified {NUM_EXPERTS} experts ({target_proj})")
+
+    print(f"Modified {total_modified} weight matrices")
+
+    # Save
+    model.save_pretrained(output_path, safe_serialization=True)
+    tokenizer.save_pretrained(output_path)
+    print(f"Saved to {output_path}")
+    print("NOW RUN EVAL to verify this improved quality. If not, revert.")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", required=True, help="Path to merged model")
+    parser.add_argument("--output", required=True, help="Output path")
+    parser.add_argument("--keep-fraction", type=float, default=0.95,
+                        help="Fraction of top singular values to KEEP (default: 0.95 = drop bottom 5%%)")
+    parser.add_argument("--layer", type=int, default=None,
+                        help="Specific layer to target (default: all layers — NOT recommended)")
+    parser.add_argument("--proj", default="gate_proj", choices=["gate_proj", "up_proj"],
+                        help="Projection to target (default: gate_proj)")
+    args = parser.parse_args()
+
+    apply_laser_selective(args.model, args.output, args.keep_fraction, args.layer, args.proj)
+```
+
+---
+
+## Reference: Key Papers Cited
+
+| Paper | arXiv | Used For |
+|---|---|---|
+| ESFT (DeepSeek) | 2407.01906 | Expert profiling methodology, shared expert freeze evidence |
+| HELLoRA | 2605.18795 | Primary training method |
+| Unified Expert Scoring | 2606.15716 | MAN scoring > frequency scoring for expert importance |
+| DoRA | 2402.09353 | LoRA upgrade (+1-4% accuracy) |
+| LASER | 2312.13558 | Post-training rank reduction |
+| EAQuant | 2506.13329 | QLoRA breaks MoE routing (do not use) |
+| Geometric coupling (routers) | 2605.12476 | Do not fine-tune routers |
+| MoE-Sieve | 2603.24044 | Routing-guided LoRA validation |
+| DR-LoRA | 2601.04823 | Dynamic rank allocation (future iteration) |
+| OGPSA v3.1 | ogpsa_persona_validation_v3.1.md | Personality geometry capture |
+| Training best practices | training_pipeline_best_practices.md | alpha=2r, checkpoint averaging, OPLoRA |
+
+---
+
+## Changelog
+
+- **v2.1** (July 2026): Applied 12-point red-hat review fix (daimon_pipeline_v2_redhat.md). Fixes: (1) LASER SVD direction inverted -- moved to optional offline experiment; (2) Phase 3 checkpoint loading via PeftModel.from_pretrained with is_trainable=True + weight-delta assert; (3) Added stratified eval split to fix eval_strategy crash; (4) Replaced Unsloth throughput estimates with realistic raw-PEFT speeds, added 50-step throughput probe gate and max_steps circuit breaker; (5) Disk preflight gate increased from 150 GB to 400 GB; (6) VRAM estimates corrected from 55-65 GB to 90-115 GB peak; (7) TruthfulQA removed from eval battery (contaminated -- in training data 15x); (8) Added checkpoint egress rsync after every save; (9) Pinned dependency versions, torch installed first; (10) Added throughput probe as hard budget gate; (11) Added backward-pass smoke test for gradient flow verification; (12) Budget table corrected to $30-45 primary / $52-67 reserve.
+- **v2.0** (July 2026): Complete rewrite for H200 cloud training. HELLoRA method selection. Budget-constrained design. SOTA sweep integration.
+- **v1.0** (June 2026): Original spec targeting Apple Silicon (Margaret). 8-stage pipeline with DPO and RL. See `daimon_training_spec.md`.