decoding_grid_report.md

# Comparative 1000-token decoding grid — medium checkpoint selection

## Executive verdict

Recommended candidate for `1gpu-llm-medium`:

- checkpoint: `step_34200`
- preset: `creative`
- tuning score: `3.1683`
- holdout score: `3.8405`
- EOS termination: `100%` on tuning and holdout
- truncation at 1000 tokens: `0%`
- loop rate: `0%` on both splits
- repeated 4-gram rate: `0%` tuning, `25%` holdout
- mean generated length: `410.6` tuning / `494.8` holdout tokens
- median generated length: `392` tuning / `582.5` holdout tokens
- distinct-2: `0.9531` tuning / `0.9536` holdout
- language switches: `0%` on both splits

This is the strongest overall combination because it is the only checkpoint/preset
pair that combines the scalar champion checkpoint, a tuning winner confirmed by the
holdout, long completions, zero truncation, zero loops, high diversity, and no
language switching.

Conservative alternative:

- checkpoint: `step_34000`
- preset: `anti_loop`
- tuning score: `3.4540`
- holdout score: `3.6036`
- EOS termination: `100%` on both splits
- truncation: `0%`
- loop rate: `0%` on both splits
- repeated 4-gram rate: `0%` tuning, `25%` holdout
- mean length: `183.0` tuning / `265.5` holdout

It is cleaner against repetition but produces shorter, more conservative answers
and remains behind `step_34200 + creative` on the holdout score.

The behavior champion `step_34800` is not promoted. Its tuning winner is `creative`,
but holdout selects `balanced`; balanced is very short (`141.8` tokens mean) and
therefore is not rewarded as a final choice merely for stopping early. `step_34800`
does not provide a robust behavior advantage over `step_34200` under this 1000-token
comparison.

## Experimental controls

- Checkpoints: `step_34000`, `step_34200`, `step_34800`
- Presets: `anti_loop_conservative`, `anti_loop`, `balanced`, `creative`
- `max_new_tokens`: `1000` for every preset
- Tuning prompts: 7
- Holdout prompts: 4, disjoint from tuning
- Seed: `1337`
- Tokenizer: `/mnt/apps/llm-nanochat/tokenizers/tokenizer_20260515_en50it50_webwiki_stratified_500M`
- Device/dtype: CUDA / `bf16`
- Generation count: 7 per preset on tuning and 4 per preset on holdout, one seed
- Early stopping: only the model EOS path; no artificial stop was introduced
- Holdout coverage: all four presets were evaluated, not only the tuning top-k

The repo runner does not serialize an explicit `terminated_with_eos` boolean. Since
the generator has only two exits — EOS or the `max_new_tokens` loop limit — this
report derives EOS/truncation as follows:

- `num_generated_tokens < 1000`: EOS termination
- `num_generated_tokens == 1000`: truncation at the configured limit

## Complete tuning table

`rep` is repeated-4gram rate; `loop` is the stricter repeated-4gram loop rate.
Higher EOS, distinct-1/2 and language consistency are better; lower rep/loop and
switch rates are better.

| checkpoint | preset | score | EOS | trunc. | mean | median | distinct-1 | distinct-2 | rep | loop | lang. consistency |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 34000 | anti_loop_conservative | 2.9462 | 71.4% | 28.6% | 351.0 | 125.0 | 0.4763 | 0.8471 | 42.9% | 0% | 0.9848 |
| 34000 | anti_loop | **3.4540** | 100% | 0% | 183.0 | 183.0 | 0.5750 | 0.9567 | 0% | 0% | 0.9262 |
| 34000 | balanced | 2.0965 | 100% | 0% | 312.1 | 256.0 | 0.4833 | 0.8582 | 71.4% | 14.3% | 0.9286 |
| 34000 | creative | 2.3089 | 85.7% | 14.3% | 299.4 | 220.0 | 0.5312 | 0.9237 | 28.6% | 14.3% | 0.9203 |
| 34200 | anti_loop_conservative | 3.0936 | 85.7% | 14.3% | 265.9 | 66.0 | 0.4779 | 0.8545 | 14.3% | 0% | 0.7850 |
| 34200 | anti_loop | 3.0177 | 100% | 0% | 268.0 | 243.0 | 0.5552 | 0.9439 | 0% | 0% | 0.9259 |
| 34200 | balanced | 2.5360 | 100% | 0% | 202.7 | 146.0 | 0.5922 | 0.9508 | 28.6% | 0% | 0.9286 |
| 34200 | creative | **3.1683** | 100% | 0% | 410.6 | 392.0 | 0.5119 | 0.9531 | 0% | 0% | 0.9339 |
| 34800 | anti_loop_conservative | 2.5619 | 85.7% | 14.3% | 279.3 | 130.0 | 0.5378 | 0.7822 | 28.6% | 14.3% | 0.8571 |
| 34800 | anti_loop | 2.5392 | 85.7% | 14.3% | 258.9 | 133.0 | 0.5580 | 0.9134 | 28.6% | 0% | 0.9259 |
| 34800 | balanced | 2.6047 | 100% | 0% | 98.1 | 57.0 | 0.5764 | 0.9290 | 14.3% | 0% | 0.9286 |
| 34800 | creative | **2.8813** | 100% | 0% | 323.4 | 95.0 | 0.5315 | **0.9591** | 14.3% | 0% | 0.9263 |

Tuning winners:

- `step_34000`: `anti_loop`
- `step_34200`: `creative`
- `step_34800`: `creative`

## Complete holdout table

| checkpoint | preset | score | EOS | trunc. | mean | median | distinct-1 | distinct-2 | rep | loop | lang. consistency |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| 34000 | anti_loop_conservative | 3.4012 | 100% | 0% | 129.5 | 110.5 | 0.6005 | 0.9024 | 25% | 0% | 1.0000 |
| 34000 | anti_loop | **3.6036** | 100% | 0% | 265.5 | 199.5 | 0.6262 | 0.9524 | 25% | 0% | 1.0000 |
| 34000 | balanced | 3.1137 | 100% | 0% | 205.0 | 195.0 | 0.5928 | 0.9180 | 50% | 0% | 0.9886 |
| 34000 | creative | 3.3935 | 100% | 0% | 322.0 | 358.0 | 0.6184 | **0.9644** | 0% | 0% | 1.0000 |
| 34200 | anti_loop_conservative | 3.0267 | 100% | 0% | 142.0 | 127.5 | 0.6136 | 0.8899 | 50% | 0% | 1.0000 |
| 34200 | anti_loop | 3.5983 | 100% | 0% | 285.0 | 351.5 | 0.6158 | 0.9389 | 25% | 0% | 0.9931 |
| 34200 | balanced | 2.5511 | 100% | 0% | 157.5 | 112.0 | **0.6856** | 0.9550 | 0% | 0% | 1.0000 |
| 34200 | creative | **3.8405** | 100% | 0% | **494.8** | **582.5** | 0.5777 | 0.9536 | 25% | 0% | 1.0000 |
| 34800 | anti_loop_conservative | 3.5655 | 75% | 25% | 348.0 | 192.0 | 0.5101 | 0.8564 | 25% | 0% | 0.9975 |
| 34800 | anti_loop | 3.5889 | 100% | 0% | 224.0 | 184.0 | 0.6499 | 0.9696 | 25% | 0% | 1.0000 |
| 34800 | balanced | **3.8743** | 100% | 0% | 141.8 | 149.5 | 0.6645 | 0.9621 | 0% | 0% | 1.0000 |
| 34800 | creative | 2.8716 | 100% | 0% | 179.0 | 159.0 | **0.6790** | 0.9591 | 25% | 0% | 1.0000 |

Holdout winners by checkpoint:

- `step_34000`: `anti_loop`
- `step_34200`: `creative`
- `step_34800`: `balanced`

The `step_34200` tuning winner is confirmed by holdout. The `step_34800` tuning
winner is not confirmed: holdout prefers `balanced`, but that preset terminates at
only 141.8 tokens on average. That shortness is treated as a weakness, not a bonus.

## First degeneration / loop analysis

The first-loop metric is conservative: it reports the first generated word position
where a 4-gram has appeared three times. When no such event occurs, the value is
`none`; first repeated-4gram position is also tracked separately.

### Tuning observations

- `step_34000 + anti_loop`: no repeated 4-gram and no loop across all 7 samples.
- `step_34200 + creative`: no repeated 4-gram and no loop across all 7 samples.
- `step_34800 + creative`: one repeated 4-gram sample, but no strict loop.
- `step_34000 + balanced`: first strict loop at token 41 in one sample; repeated-4gram rate 71.4%.
- `step_34000 + creative`: first strict loop at mean token 142 in one sample.
- `step_34800 + anti_loop_conservative`: one strict loop, first occurring at token 248.

### Holdout observations

No holdout configuration produced a strict loop (`loop_rate=0%`). Repeated 4-grams
remain in several cases, especially `step_34200 + creative` and the anti-loop
variants, but they occur without reaching the stricter three-repetition threshold.

## Qualitative inspection

Representative long generations were inspected for factuality, relevance, language
stability and late degeneration.

- `step_34000 + anti_loop` is the most controlled: it reaches EOS consistently and
  avoids strict loops, but often produces conservative encyclopedic continuations
  with factual drift (for example, Paris geography and historical details).
- `step_34200 + creative` produces the longest useful continuations. It stays in the
  requested language and avoids strict loops, but factual/semantic drift appears in
  long completions. The issue is quality drift, not a decoding collapse.
- `step_34800 + creative` is lively and diverse, but is less stable as a checkpoint
  choice: tuning and holdout select different presets, and the median tuning length
  is only 95 tokens despite a 1000-token allowance.
- `balanced` on `step_34800` wins the raw holdout score mainly with short, clean
  completions. It is rejected as the final preset because early EOS must not be
  mistaken for superior long-form behavior.

No preset systematically truncates on the selected final pair `step_34200 + creative`.
The main remaining limitation is factual/semantic degradation in long text, not
EOS handling, language switching or strict repetition loops.

## Final operational recommendation

### Definitive candidate

Keep and use:

```text
checkpoint: /mnt/apps/llm-nanochat/checkpoints/20260715_resume-gpt2medium-gpt2preln-k20-optimizeronly-cpt14700-step34000-d1700-webwiki/step_34200.pt
preset: creative
```

Reason: tuning winner confirmed by holdout, zero truncation, zero strict loops,
longest useful completions, high distinct-2, and stable EN/IT language behavior.

### Behavior-oriented alternative

Keep as a conservative fallback:

```text
checkpoint: /mnt/apps/llm-nanochat/checkpoints/20260703_continual-pretraining-gpt2medium-gpt2preln-k20-step14700-lr5e5-w500-s18500-d2000-final1e5-webwiki/step_34000.pt
preset: anti_loop
```

This pair is the most robust against repetition and is confirmed by holdout, at the
cost of shorter and less expressive completions.

### Retention and deletion candidates

Retain now:

1. `step_34200.pt` — definitive candidate with `creative`
2. `step_34000.pt` — parent/reference and conservative fallback with `anti_loop`
3. `step_34800.pt` — behavior experiment retained until the final release decision

Potentially eliminable only after explicit confirmation:

- `step_34800.pt`, if the project keeps only the definitive candidate plus parent
  reference.

No checkpoint was deleted by this operation. The other 34k-tail checkpoints were
not part of this three-candidate comparison and are outside this cleanup decision.

## Raw artifacts

- Config:
  `configs/eval/20260716_medium_checkpoint_decoding_grid_1000.yaml`
- Output root:
  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000`
- Parent output:
  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/parent_step34000`
- Scalar output:
  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/scalar_step34200`
- Behavior output:
  `/mnt/apps/llm-nanochat/evals/20260716_medium_checkpoint_decoding_grid_1000/behavior_step34800`
- Launch log:
  `/mnt/apps/llm-nanochat/launch_logs/20260716_135423_decoding_grid_medium_three_checkpoints_1000.log`