decoding_report.md

# Decoding search report — `decoding_tuning_en_it_v1`

## Scope

- checkpoint: `/mnt/apps/llm-nanochat/checkpoints/20260622_resume-gpt2small-gpt2preln-k20-wsds800-final2e5-webwiki-step8000-dense50/step_8600.pt`
- tokenizer: `/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M`
- tuning prompts: `/tmp/llm-nanochat-dev-p0p1/eval_prompts/decoding_tuning.jsonl`
- holdout prompts: `/tmp/llm-nanochat-dev-p0p1/eval_prompts/decoding_tuning_holdout.jsonl`
- seeds: `[1337, 1338, 1339]`
- holdout top-k: `3`

This report intentionally separates **decoding tuning** from the checkpoint-selection benchmark. The tuning prompt suite is the only data used to choose the recommended preset; holdout results are shown only as a confirmation check.

## Tuning leaderboard

| Rank | Preset | Score | Pass rate | Distinct-2 | Loop rate | Repeated 4-gram | Lang. consistency | Length closeness | Avg tokens | Decoding |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | balanced | 2.789 | 0.0833 | 0.9176 | 0 | 0.0952 | 0.9514 | 0.9487 | 48.5714 | temp=0.8 top_k=50 top_p=0.95 rep=1.1 ng=0 max_new=64 |
| 2 | creative | 2.7751 | 0.0833 | 0.9565 | 0.0476 | 0.0952 | 0.9792 | 0.9286 | 54.8571 | temp=1 top_k=100 top_p=0.95 rep=1.1 ng=0 max_new=64 |
| 3 | anti_loop | 2.7175 | 0 | 0.9261 | 0 | 0.0476 | 0.9583 | 0.9654 | 49.4286 | temp=0.8 top_k=50 top_p=0.9 rep=1.15 ng=4 max_new=64 |
| 4 | conservative | 1.3666 | 0.25 | 0.6885 | 0.2857 | 0.6667 | 0.9353 | 0.9635 | 49.3333 | temp=0.6 top_k=40 top_p=0.9 rep=1.05 ng=0 max_new=64 |
| 5 | greedy | -0.9418 | 0 | 0.055 | 0.7143 | 0.7143 | 0.7708 | 0.971 | 49.7143 | temp=0 top_k=0 top_p=1 rep=1 ng=0 max_new=64 |

## Holdout confirmation

Holdout prompts are only used after tuning has already chosen candidates. They are not part of the tuning leaderboard.

| Rank | Preset | Score | Pass rate | Distinct-2 | Loop rate | Repeated 4-gram | Lang. consistency | Length closeness | Avg tokens |
|---|---|---|---|---|---|---|---|---|---|
| 1 | anti_loop | 3.3392 | 0.3333 | 0.9385 | 0 | 0.0833 | 1 | 0.8428 | 59.25 |
| 2 | balanced | 3.1885 | 0.3333 | 0.9244 | 0 | 0.1667 | 0.9762 | 0.8607 | 58.3333 |
| 3 | creative | 3.096 | 0.1111 | 0.9579 | 0 | 0 | 1 | 0.9486 | 53.8333 |

## Recommendation

- recommended preset: `balanced`
- tuning score: `2.789`
- holdout checked: `True`
- holdout results should be used as confirmation, not as the tuning objective.

## Notes

- Large grids should be entered only after a coarse pass has identified a sensible band.
- This workflow is intended for checkpoint-specific decoding tuning, not for declaring the best checkpoint itself.