Generation settings: replace guessed defaults with swept, measured recommendations

README.md

@@ -68,7 +68,8 @@ with torch.no_grad():
     out = model.generate(
         **inputs,
         max_new_tokens=256,
-        do_sample=False,           # greedy; see Generation settings below
+        do_sample=False,           # greedy is fine for a short answer like this;
+                                   # for open chat use the sampling settings below
         eos_token_id=5,            # <|im_end|> -- also the repo default
         pad_token_id=3,            # <pad>
     )
@@ -93,16 +94,61 @@ The model uses ChatML. `tokenizer.apply_chat_template` applies it for you; the r
 {assistant}<|im_end|>
 ```
 
-A system turn is optional. Uzbek-language system prompts work best — that is what the
-model was trained with.
+A system turn is optional, and for general chat you should leave it out — a generic
+system prompt measurably increases repetition (see Generation settings). Task-specific
+system prompts, in Uzbek, work well.
 
 ### Generation settings
 
+These are measured, not guessed: 24 Uzbek chat prompts across 11 categories, 2–3 seeds
+per configuration, scored automatically for verbatim sentence repetition, failure to
+emit `<|im_end|>` within the token budget, and script leakage.
+
+**Recommended for open chat:**
+
+```python
+out = model.generate(
+    **inputs,
+    max_new_tokens=512,
+    do_sample=True,
+    temperature=0.7,
+    top_p=0.9,
+    repetition_penalty=1.10,   # not optional -- see below
+    use_cache=True,
+)
+```
+
 | setting | value | why |
 |---|---|---|
-| `eos_token_id` | **5** (`<\|im_end\|>`) | The turn terminator, and already the default in `config.json` / `generation_config.json` — you do not need to pass it. Do **not** override it with the pretraining EOS (`</s>`), which never appears in chat data: generation would then run to `max_new_tokens`. |
-| `do_sample` | `False` for classification/extraction; `True`, `temperature≈0.7`, `top_p≈0.9` for open chat | `generation_config.json` ships `do_sample: true` with no temperature or top_p set, so pass these explicitly. Every benchmark below was measured greedy. |
-| `dtype` | `torch.bfloat16` | Trained in bf16. |
+| `repetition_penalty` | **1.05–1.10** for chat | The single most important setting. Without it the model restates whole sentences verbatim. Measured duplicate-sentence rate at temperature 0.7: **4.7% at `1.00`, 2.1% at `1.02`, 0.0% at `1.05` and above.** |
+| `do_sample` / `temperature` / `top_p` | `True`, `0.7`, `0.9` for chat; `False` (greedy) for classification, extraction and short answers | `generation_config.json` ships `do_sample: true` with **no** `temperature` or `top_p`, so the unconfigured default is temperature 1.0 / top_p 1.0 — pass these explicitly. Terse tasks showed a 0% repetition rate under every configuration tested, so greedy is safe there. |
+| `eos_token_id` | **5** (`<\|im_end\|>`) | The turn terminator, already the default in `config.json` / `generation_config.json` — you do not need to pass it. Do **not** override it with the pretraining EOS (`</s>`), which never appears in chat data: generation would then run to `max_new_tokens`. |
+| system prompt | **omit it** for general chat | A generic system turn measurably degrades output. Duplicate-sentence rate over the same prompts: **0.0% with no system prompt, 1.9% with a generic Uzbek one, 5.2% with a generic English one** (at temperature 0.7, `repetition_penalty` 1.05); without a repetition penalty the same comparison is 11.2% / 23.7% / 14.9%. Task-specific system prompts (a required format, a persona) are fine — it is the generic "you are a helpful assistant" turn that hurts. |
+| `dtype` | `torch.bfloat16` | Trained in bf16. `float16` is also safe — no overflow, and output quality is indistinguishable — so pre-Ampere GPUs are supported. `float32` doubles memory for **half** the throughput (205 vs 412 tok/s) and changes nothing. |
+
+**Greedy decoding degrades as the output gets longer**, which is why it is recommended
+above only for short outputs. On chat and long-form prompts with a 768-token budget:
+
+| configuration | never emits `<\|im_end\|>` | duplicate sentences | worst case |
+|---|---:|---:|---:|
+| greedy | 23.1% | 27.3% | one sentence repeated **9.8×** |
+| `t=0.7, top_p=0.9` | 15.4% | 8.0% | 2.2× |
+| `t=0.7, top_p=0.9, rp=1.05` | 11.5% | 3.0% | 1.7× |
+| **`t=0.7, top_p=0.9, rp=1.10`** | **0.0%** | **0.8%** | **1.1×** |
+
+Lowering the temperature makes this worse, not better: temperature 0.3 was the worst
+configuration measured (19.1% duplicate sentences), because sharpening the distribution
+locks the model into the repeat loop. Determinism is genuinely in tension with quality
+here: greedy plus `repetition_penalty=1.10` still leaves 7.9% duplicate sentences —
+better than greedy alone, but far short of sampling. If you need reproducible output,
+sample with a fixed seed rather than decoding greedily.
+
+Uzbek Cyrillic prompting is the weakest case — the highest truncation and repetition
+rates of any category — so raise `max_new_tokens` and keep the repetition penalty on.
+
+Batch size changes greedy output: identical prompts decoded at batch 1 and batch 12
+matched in only 24 of 32 cases, because left-padding shifts the numerics. Fix the batch
+size when comparing runs.
 
 Memory: the checkpoint is 11.0 GB on disk (embeddings and `lm_head` are stored fp32); loading with
 `dtype=torch.bfloat16` as above casts them down to ~10.3 GB of weights, so a single 16 GB GPU is