data/eval/sft_cleanup_report.md

# SFT Cleanup Report

Date: 2026-06-17

## What changed

### 1. Main train expanded
- Merged **60** `bedimcode_portfolio` rows from `datasets/bedimcode_portfolio_converted.jsonl`
- Main train: **1,472 rows** (was 1,412)

### 2. SFT builder fixed
- `scripts/build_sft_messages.py` now reads **`datasets/mythos_coder_train.jsonl`** by default
- Optional `--converted-dir` kept for partial rebuilds
- Assistant messages compressed:
  - Numbered solutions → max 4 bullets
  - Verification → max 3 checks
  - Investigation steps capped at 4, each truncated
  - Section char limits on Diagnosis / Plan / Implementation / Verification / Lesson

### 3. Template audit
- New script: `scripts/audit_sft_templates.py`
- Report: `data/eval/sft_template_audit.md`

### 4. SFT rebuilt
- Output: `data/train/mythos_sft_messages.jsonl` — **1,472 rows**
- Synced to `spaces/mythos-coder-zerogpu/data/train/mythos_sft_messages.jsonl`

## Before vs after (SFT assistant length)

| Metric | Before cleanup | After cleanup |
|--------|---------------:|--------------:|
| SFT rows | 1,100 | 1,472 |
| Source | `data/converted/` only | full main train |
| Avg assistant chars | ~2,522 | **1,659** |
| Missing from SFT | 372+ rows | 0 |

## Remaining dataset debt

- **93.8%** of raw solutions use numbered-list templates (mostly game batch)
- **1,000** game rows start solutions with `Scan ...` (echo of investigation steps)
- Future fix: regenerate game raw rows with prose solutions, not step echoes

## Next steps

1. Push Space update: `cd spaces/mythos-coder-zerogpu && git add data/train/mythos_sft_messages.jsonl && git commit && git push`
2. Retrain LoRA on ZeroGPU with the new 1,472-example SFT file
3. Re-run eval with `scripts/test_lora_model.py` and compare to `data/eval/mythos_lora_eval_report.md`