data/eval/sft_template_audit.md

# SFT Template Audit

Train file: `D:\mythos-coder-data\datasets\mythos_coder_train.jsonl`
SFT file: `D:\mythos-coder-data\data\train\mythos_sft_messages.jsonl`

## Dataset coverage

- Total train rows: **1472**
- Avg user prompt length: **157** chars
- Avg raw solution length: **580** chars

### Rows by source bucket

- `other`: 813
- `game_repo_batch`: 400
- `bedim_restaurant`: 100
- `html5up`: 99
- `bedim_portfolio`: 60

## Template issues (raw train)

- Numbered-list solutions: **1381** (93.8%)
- Solutions with 5+ numbered steps: **1151**
- Verification rows matching fake/browser boilerplate: **2**

### Repetitive solution openings (game batch pattern)

- `Scan ...`: 1000

### Most repeated failure_log prefixes


### Most repeated lesson prefixes


## Build-time mitigations

- `build_sft_messages.py` now reads from `datasets/mythos_coder_train.jsonl`.
- Assistant responses are compressed: numbered solutions capped to 4 bullets, verification trimmed to 3 checks.
- Diagnosis drops redundant `Initial problem:` prefix and limits investigation steps to 4.

## SFT output after compression

- SFT rows: **1472**
- Avg assistant message: **1659** chars
- Max assistant message: **1951** chars
- Rows still over 1800 chars: **811**

## Recommendations

1. Regenerate game-repo raw rows with shorter `solution` prose instead of echoing investigation steps.
2. Replace screenshot/recording verification text with concrete command or browser checks.
3. Keep user prompts messy/vague in eval only; train prompts should stay specific.
4. Retrain LoRA after SFT rebuild and re-run `test_lora_model.py`.