# GPT2-small decoding-grid comparison: step_8600 vs step_18000 vs step_18800

Date: 2026-07-03

## Scope

This note compares the completed GPU decoding-grid artifacts for:

- `step_8600`
- `step_18000`
- `step_18800`

The comparison focuses on:

- Italian behavior
- loop / repetition resistance
- cloze / factual prompts
- free EN / IT prompts

The tuning split is the main grid-selection signal. The holdout split is a confirmation check.

## Executive reading

- **Checkpoint-level verdict:** `step_18800` remains the benchmark / loss winner.
- **Preset-level verdict:** `step_18000 + anti_loop` is the strongest overall decoding-grid pairing once tuning and holdout are both considered.
- **Why that split matters:** the raw tuning winner is `step_18800 + anti_loop`, but the raw holdout winner is `step_18000 + anti_loop`, and the average of the two split scores favors `step_18000 + anti_loop`.

## Score table

| Checkpoint | Tuning winner | Tuning score | Holdout winner | Holdout score | Tuning+holdout avg | Read |
|---|---|---:|---|---:|---:|---|
| `step_8600` | `balanced` | 2.789 | `balanced` | 3.188 | 2.989 | stable, but weakest overall |
| `step_18000` | `creative` | 3.370 | `anti_loop` | 3.927 | 3.496 | best combined grid result |
| `step_18800` | `anti_loop` | 3.034 | `anti_loop` | 3.210 | 3.122 | consistent, but below `step_18000` on the combined read |

## Category notes

### Italian behavior

- `step_8600` is the weakest on Italian factual prompts, but it still occasionally lands the right answer under `balanced`.
- `step_18000` improves English factual completion materially and keeps Italian free-form output coherent, but it still misses the Italian cloze answer in the best zero-loop setting.
- `step_18800` is the most visibly polished in style, but it regresses on the Italian cloze prompt: it keeps drifting into the wrong factual chain instead of producing `Roma`.

### Loop / repetition

- `greedy` is unusable on all three checkpoints.
- `anti_loop` is the only preset that is consistently safe on repetition across the sweep.
- `step_18000 + anti_loop` is the best compromise: no loops on either split and the strongest combined score.
- `step_8600` is the most repetition-sensitive checkpoint overall.

### Cloze / factual prompts

- On the English cloze prompt, `step_18000` is the strongest of the three checkpoints.
- On the Italian cloze prompt, none of the checkpoints is fully reliable, but `step_18800` is the most obviously wrong because it settles into the wrong entity chain rather than a near-miss.
- The factual prompts are the clearest place where benchmark loss and decode quality diverge: `step_18800` still wins on loss, but it does not produce the best factual decoding behavior.

### Free EN / IT prompts

- All three checkpoints mostly preserve language identity on the free-form prompts.
- The main distinction is robustness:
  - `step_8600` is coherent but bland and more repetition-prone.
  - `step_18000 + anti_loop` is the cleanest balanced setting for free EN / IT generation.
  - `step_18800` reads smoother, but its factual slips keep it from being the best overall operational choice.

## Tuning / holdout mismatch

- `step_8600` is internally stable: tuning and holdout both prefer `balanced`.
- `step_18000` is the clearest mismatch: tuning prefers `creative`, holdout prefers `anti_loop`.
- `step_18800` is internally consistent on the best preset (`anti_loop`), but the checkpoint still underperforms `step_18000 + anti_loop` on the combined read.

## Bottom line

- **Checkpoint winner:** `step_18800`
- **Best decoding-grid pairing:** `step_18000 + anti_loop`
- **Main tradeoff:** `step_18800` is the best loss checkpoint, but `step_18000 + anti_loop` is the safer decode choice for Italian + free-form usage in this sweep.