# Reproduction Guide — GLM-5.2 Ablation Research / Project AESOP

## Prerequisites
- 8× H200 SXM (141GB each) or equivalent (~1130GB total GPU memory)
- GLM-5.2 FP8 base weights (HuggingFace or local)
- Fable 5 training data: fable5-chatml.jsonl (4,876 examples, 50MB)
- Refusal PCA directions: refusal_pca.pt (2.9MB, 41 layers × 3 PCA components × 6144 hidden dim)
- Layer indices: layer_indices.json (layers 25-65)
- AdvBench prompts: advbench.csv (81KB, 100 test prompts)
- Python 3.10+, PyTorch 2.x, transformers, peft, datasets (HuggingFace)

## Archived Artifacts (this package)
```
artifacts/
  2e-snapshot/          # Full workspace snapshot from 2E (Vast.ai 8×H200)
    *.py                # All scripts (63 files)
    *.log               # All training/benchmark logs (80 files)
    bench_*/            # Per-model benchmark results (18 dirs)
    refusal_pca.pt      # PCA refusal directions
    layer_indices.json  # Target layers for PCA extraction
    fable5-chatml.jsonl # Training data (4,876 examples)
    advbench.csv        # AdvBench evaluation prompts
    pipeline-status.txt # Pipeline state timeline
  test3a/               # Test 3a model card + benchmark results
  aesop/                # AESOP model card + benchmark results  
  fable5_r2/            # Fable5-R2 model card + benchmark results
  COMPARISON.md         # Cross-model comparison table with harness notes
  STATISTICAL_ANALYSIS.md # Wilson CIs + significance tests
benchmarks/             # Unified benchmark harness (v3 — canonical)
  harness.py            # All 9 benchmarks, consistent scoring
  run.py                # CLI runner
  stats.py              # Statistical utilities
  test_harness.py       # Scoring function unit tests
  README.md             # Usage + methodology + known limitations
```

## Step 1: Extract Refusal Directions
```bash
python3 extract_refusal_directions.py \
  --model /path/to/glm52-fp8 \
  --output refusal_pca.pt \
  --layers 25,26,...,65
```
Expected output: refusal_pca.pt (~2.9MB), 41 layers × 3 PCA components × 6144

## Step 2A: Test 3a (ablation hooks during training)
```bash
python3 train_ablated_fable5_lora.py \
  --model /dev/shm/glm52-test3a-merged \
  --data fable5-chatml.jsonl \
  --out /workspace/checkpoints/test3a \
  --ablation_layers 62,63,64,65 \
  --ablation_coeff 0.1 \
  --rank 64 --alpha 128 --lora_min_layer 60 \
  --max_seq_len 2048 --lr 2e-5 --warmup 10 \
  --max_steps 610 --save_every 100 --log_every 1 \
  --seed 42
```
Expected: ~9.5h on 8×H200, first_loss ~1.29, final_loss ~1.18

## Step 2B: Fable5-R2 (no hooks during training)
Same as 2A but WITHOUT --ablation_layers / --ablation_coeff (or set coeff to 0).

## Step 2C: AESOP (same as 2A — this IS the AESOP full training)
Identical config to Step 2A.

## Step 3: Merge
```bash
python3 merge_aesop.py  # or merge_lora_bf16.py
```

## Step 4: Serve + Benchmark
```bash
# Serve with vLLM
vllm serve /path/to/merged/model --tensor-parallel-size 8 --max-model-len 4096

# Run unified harness
python3 benchmarks/run.py --model <model-name> --output bench_<variant>/
```

## Step 5: Compare Results
```bash
python3 benchmarks/stats.py --compare bench_test3a/ bench_aesop/ bench_fable5_r2/
```

## Known Issues
1. The harness versions were inconsistent in the original research (v1 vs v2). Use ONLY benchmarks/harness.py (v3) for clean comparisons.
2. MMLU-Pro and GSM8K sample n=100 — too small for 5pp significance. See STATISTICAL_ANALYSIS.md.
3. Test 3a's exact training command was not logged at the time. The config in this guide is reconstructed from the AESOP full training log (same approach, same base, same LoRA config).
4. The step-0 baseline (raw ablated base with no LoRA) was not run. This baseline is important for establishing clean separation between base-model ablation and LoRA effects.

## Semantic Version
Artifact package v1.0 — created 2026-06-23