| # Teacher bake-off — Model G (solution generator) |
|
|
| Data-driven teacher selection. 40 NuminaMath-CoT problems, one solution each, |
| graded by boxed-answer equivalence (`scripts/teacher_bakeoff.py`). |
|
|
| scripts/teacher_bakeoff.py --models gpt-5.6-sol claude-opus-4-8 claude-sonnet-5 --n 40 |
| |
| | model | keep-rate | median tok | p90 tok | % over 2048 | sec/ex | |
| |---|---|---|---|---|---| |
| | **claude-opus-4-8** | **57%** | 155 | 703 | 0% | 6.7 | |
| | gpt-5.6-sol | 32% | 103 | 324 | 0% | 10.4 | |
| | claude-sonnet-5 | 12% | 224 | 691 | 0% | 9.9 | |
|
|
| **Winner: `claude-opus-4-8`** — highest keep-rate (most usable data per call), |
| fastest, and traces fit the 4096-token student ceiling with room to spare. |
|
|
| `claude-fable-5` was requested but **excluded**: 403 on both routes (AWS Bedrock |
| Marketplace access not granted on this gateway account). |
|
|
| **Caveats:** n=40 is noisy; absolute keep-rates are depressed by NuminaMath |
| difficulty + strict grading + format adherence, not just model capability — the |
| *relative* ranking is the signal. For the real G build, rejection sampling with |
| `--max-attempts > 1` lifts the effective keep-rate. |
|
|
| ## Teacher assignments |
| - **Model G** (solution generator): `claude-opus-4-8`. |
| - **Model V** (critique writer; label already known from PRM800K → easier job): |
| `claude-haiku-4-5` (cheaper) is sufficient; use `claude-opus-4-8` if budget allows. |
|
|
