NOTES.md

# NOTES — diversity-aware post-training for creative writing

**Everything learned, every bug, every decision. Written so a new session can
resume cold.** Last updated: 2026-08-17 ~02:00 IST (Sun night / Mon morning).

---

## 0. TL;DR — where the project stands

**The core hypothesis is supported.** A quality-gated pairwise-deviation reward
(E1) moved semantic diversity **5-6x further** than quality-only GRPO (E0), at a
cost of 0.06 judge points.

| metric (first vs last quarter, 300 steps, 4800 stories/arm) | E0 quality-only | E1 +deviation |
|---|---|---|
| judge quality | +0.303 | +0.243 |
| **mean pairwise deviation** | +0.0037 | **+0.0200** |
| **group log-det volume** | +0.165 | **+0.935** |
| story length (words) | **+30.8** | +0.7 |
| policy entropy | -2.1% | -3.1% |
| gate pass | -0.005 | +0.0008 |
| frac_groups_degenerate | 0.0067 | **0** |

**Still running / not yet done:** E2 (div-grpo-group, adds log-det marginal),
E4a/E4b (DivPO emb/prob), E3 (multi-positive weighted DPO), the 7-model eval
harness, the final frontier report, and the whole 8B scaling arm.

---

## 1. The single most important context

**The base model is ALREADY collapsed.** Qwen3-4B-Instruct-2507 scores
**effective rank 2.006 out of 16** across the 16,000-story pool: 16 stories for
one prompt span ~2 effective semantic directions, at ~0.87 mean cosine
similarity.

This reframes the whole study. It is **not** "does RL cause collapse" — the
model arrives collapsed. It is **"can any objective lift diversity off a floor
that pretraining/instruction-tuning already imposed."** E0 confirms the floor is
sticky (quality-only RL barely moves it); E1 shows it *can* be lifted.

**The collapse is TONAL, not lexical.** 92-95% of every story carries
solemn/elegiac vocabulary; only ~15% carries comic vocabulary — even on
explicitly comic prompts. Given "Cthulhu disappoints his constituency by failing
to deliver the promised chaos" (a joke), the base model wrote six straight-faced
atmospheric-horror pieces. It has one register and applies it universally.

Consequence: **n-gram metrics (distinct-4, self-BLEU) should separate the arms
far less than effective rank does.** Surface-diversity metrics are near-blind to
this failure. Worth checking in the final eval table — it's a falsifiable
prediction of the above.

---

## 2. Bugs found (the expensive ones first)

### 2.1 THE BIG ONE — learning rate was a full-FT rate applied to LoRA
The brief said lr 1e-6..5e-6. That is a **full fine-tuning** band. These are
LoRA adapters (r=32, alpha=64), which want **10-50x** that.

Evidence: E0 at 3e-6 ran **171 steps with KL pinned at ~0.0008 and NOT GROWING**
— the adapter was effectively frozen; every metric sat inside its noise band.
A 40-step probe at 3e-5 showed KL compounding (0.0008 -> 0.0081 by step 12,
~9x E0 at the same step) with gates still 1.000 and quality rising.

**Settled on lr = 3e-5.** Cost of the mistake: ~70 min of wasted E0 training.
**If you change anything else, keep lr at 3e-5 or higher.**

### 2.2 The judge silently returned neutral 5.0 for 162/166 calls
`deepseek/deepseek-v4-flash-0731` is a **reasoning model**. With a small
`max_tokens` it spends the entire budget on hidden `reasoning` and returns
`content: null`, finish_reason `length`. The retry wrapper converted that into a
neutral `JudgeScore(quality=5.0)`.

A judge that scores everything 5.0 looks *identical* to a working judge, and a
constant reward column means zero within-group std means **zero GRPO advantage**.
Doubly silent.

**Fix:** `"reasoning": {"enabled": false}` in the OpenRouter payload. Also 25x
cheaper and 5x faster (30 output tokens vs 764). `exclude:true` and
`effort:minimal` do NOT work — they still generate reasoning internally.

**Also added:** `judge.health()` (calls_ok / failed / empty_content / truncated /
fail_rate / last_error) and `assert_healthy()`, called before training.

### 2.3 asyncio.Semaphore bound to the wrong event loop
Created in `__init__`, but `score_many_sync()` calls `asyncio.run()` — a NEW
loop every training step. It survives step 1 then raises "bound to a different
event loop" from step 2 onward, which the retry wrapper turns into neutral 5.0s
forever. **Fix:** build the semaphore inside `score_many()`, per call.

### 2.4 The marginal-channel sign trap
`m_i = logdet(L) - logdet(L_-i) <= log(1+eps) ~ 0` is **always negative**.
Gating an ineligible sample to `0.0` would hand it the *highest* diversity credit
in its group — a reward-hacking channel of our own construction.
**Fix:** ineligible samples take the **minimum value among eligible samples**,
a rule that is correct regardless of a channel's sign convention.
Covered by `test_gated_sample_never_outranks_in_marginal_channel`.

### 2.5 The specified cluster-count metric does not work
The brief asked for "k-means, silhouette-selected k" as the mode counter.
Measured on synthetic 16-point sets:

| configuration | best k | silhouette |
|---|---|---|
| fully collapsed (noise 0.001) | 4 | 0.202 |
| fully spread (random) | 7 | 0.195 |
| two clear modes | 2 | **0.976** |

Collapsed and spread are **indistinguishable** — k-means partitions isotropic
data regardless of spread. It would have reported ~4 modes for collapsed output.
**Fix:** raised `SILHOUETTE_MIN` 0.05 -> 0.50 (making n_clusters a conservative
count of well-separated modes), and added **effective rank** (exp-entropy of the
Gram spectrum, Roy & Vetterli) as the primary continuous mode measure:
1.00 identical / 2.12 two clusters / 13.29 spread. No threshold to tune.

### 2.6 All-or-nothing judge cache
`score_many()` gathered all results before writing any to cache, so a 15,872-call
batch cached nothing until it fully completed. A crash at 99% would have thrown
away ~25 min and ~$2.50. **Fix:** `asyncio.as_completed` + cache each result the
moment it lands. Verified: re-run is 100% cache hits.

### 2.7 `save_only_model` set in YAML but never passed to GRPOConfig
Cost 505 MB of `optimizer.pt` per checkpoint (768 MB total vs 253 MB of adapter).
**Lesson: verify the setting took EFFECT, not just that the edit landed.**
I hit this class of bug twice.

### 2.8 pkill / pgrep self-match — hit FOUR times, one cost real training
`pkill -f "pattern"` matches the shell running it, because the pattern appears in
its own command line. Killed my own shell 3x (exit 144), and once killed **E0 at
step 163/300** because `pgrep -f "src/train_grpo.py"` matched a transient shell
whose PID vanished instantly, firing a wait-loop early.

**RULE: never use pattern-matching to select a process to kill. Capture the PID
explicitly, verify it with `ps -p $PID -o cmd=` and a guard, then kill by PID.**

### 2.9 DivPO rho auto-adjustment moved the threshold the WRONG WAY
Skip rate was 31.4%, driven by `no_rejected=311` (not `no_chosen=3`) — 88.8% of
stories score >=6, so hundreds of prompts had no story *below* the bar. Fixing
that needs rho to go **UP**; my code lowered it (40th percentile), which fixes
the failure mode that wasn't happening. It fired and was a no-op (6.0 -> 6.0).
**Fix:** sweep the observed quality levels and pick the threshold minimizing skip
rate. Result: rho 6.0 -> **7.0**, skip 31.4% -> **4.4%**, 686 -> 956 pairs.

### 2.10 E4 would have been undertrained, making E3-vs-E4 meaningless
956 pairs at effective batch 16 = ~60 optimizer steps; E3 gets ~449. A 7.5x
asymmetry. A validation run confirmed it empirically: loss stuck at 0.691
(= log 2, no separation), reward accuracy 0.53, margin 0.026.
**Fix:** same effective batch (1 x 8) for every DPO arm, epochs set so steps
match — E4 956x4/8 = 478 vs E3 3593x1/8 = 449. Deviates from the brief's
"1-2 epochs" deliberately, because at 1 epoch DivPO cannot learn anything.

### 2.11 Misleading auto-scaled figure axes
Effective rank moving 1.60->1.69 against a **ceiling of 6** renders as a
dramatic curve under matplotlib autoscale. Anchored the axes (eff_rank 1..N,
quality 0..10). **A flat result must look flat.**

### 2.12 Smaller ones
- Prior run's completeness check used straight quotes only; curly `"` endings
  scored 0.3 instead of 1.0.
- self-BLEU could exceed 1.0 by ~1e-10 on exact matches (smoothing); clamped.
- My judge-calibration cells A and B differed in model *and* truncation,
  confounding the measurement. Replaced with a matched cell (cell A's own
  stories cut mid-word).
- I cherry-picked a 98.7th-percentile prompt as "typical" collapse. Always check
  where an example sits in the distribution before presenting it.

---

## 3. Environment gotchas (all resolved, don't rediscover)

- **Hardware: single RTX 5090, 32.6 GB.** Not 48-56 GB. This forced everything.
- **vLLM 0.27.1 on sm_120 (Blackwell):** bundled FlashInfer misreads the CUDA
  runtime (`SM 12.x requires CUDA >= 12.9`) then rejects the card as *below*
  sm_75, killing EngineCore at warm-up.
  **Fix:** `VLLM_ATTENTION_BACKEND=FLASH_ATTN`, `VLLM_USE_FLASHINFER_SAMPLER=0`.
- **TRL 1.10 API removals:** `max_prompt_length` (GRPO *and* DPO configs) and
  `warmup_ratio` (DPO) no longer exist. `DPOConfig.loss_weights` is
  **per-loss-type, NOT per-example**.
- **`use_liger_kernel=True` routes to `compute_liger_loss`, which logs only
  `[clip_ratio, kl]` — NOT `entropy`.** Keep liger OFF if you want entropy.
  (Liger was only needed at micro-batch 2; at micro-batch 1 it isn't.)
- **8B is impossible for the online arms on 32 GB.** GRPO + colocated vLLM needs
  two resident weight copies: 16.4 + 16.4 = 32.8 GB > 32 GB before any KV cache.
  LoRA does not help — it shrinks optimizer state, not weights. **DPO needs one
  copy** (reference = adapter-disabled base) and fits at ~21 GB, which is why
  only E3/E4 scale to 8B.
- **Qwen3.5-9B rejected:** 24 `linear_attention` + 8 `full_attention` layers
  (hybrid, `mamba_ssm_dtype`), plus a vision tower, MTP head, 248k vocab. Not
  transformer-only, not text-only. `Qwen/Qwen3-8B` is the pure-transformer
  ~9B-class option (there is no `Qwen3-8B-Instruct-2507`).
- **Qwen3-8B is hybrid-thinking** — `generate.render_chat` passes
  `enable_thinking=False`, else you get `<think>` blocks instead of stories.
- CPU generation is hopeless: **1.42 tok/s** for 4B fp32 on 32 cores = 5.3 hours
  per checkpoint. Never fall back to CPU for generation.

---

## 4. Design decisions worth preserving

- **GDPO aggregation is native.** TRL 1.10's
  `multi_objective_aggregation="normalize_then_sum"` is a line-for-line
  implementation of GDPO (arXiv 2601.05242, Liu et al., NVIDIA): group-wise
  normalization **per reward channel**, then batch-wise advantage normalization.
  Consequence: **alpha/gamma weight STANDARDIZED channels**, so alpha=0.5 means
  half an SD of diversity per SD of quality. No manual rescaling of d_i needed.
- **Never use a set-level scalar as a per-sample reward.** Constant within a
  group => zero within-group std => zero advantage. This killed the prior run.
  Watch `frac_reward_zero_std` — it was 0 for every batch in E0 and E1.
- **Judge scores ONE story at a time on an absolute rubric.** Batched relative
  scoring breaks the fixed tau gate and makes cross-model eval meaningless. It
  also makes the cache effective.
- **The generation system prompt is neutral and identical across every arm**,
  including base. It says nothing about being original or varied — instructing
  the model to diversify would mask the dependent variable.
- **Nothing truncates a story.** Not generation, not the judge, not the embedder.
  Over-length is DETECTED by a gate, never chopped. `max_completion_length=1024`
  sits above the entire accept region (600-word gate ~ 780 tokens).
- **Entropy is monitored, never optimized**, and read asymmetrically: a large
  fall is strong evidence creativity is dying; a rise is only permissive.

---

## 5. Findings that surprised me

1. **The judge's own "novelty" score rose +0.52 in E0 while real diversity stayed
   flat.** The judge thinks the model got substantially fresher; it did not.
   Cleanest evidence in the study that per-story LLM scoring **cannot see
   set-level collapse** — and exactly what the prior run optimized.

2. **Whole-story embeddings can MISS verbatim opening collapse.** In E0, one
   prompt went from 6 distinct openings at base to **5 of 6 identical**
   ("The city didn't sleep.") at step 300 — while `eff_rank` and `deviation`
   both drifted slightly UP. The embedding measures thematic spread and is
   near-blind to positional convergence. **Add unique-opening-rate as a
   first-class eval metric.** Across E0's checkpoints, unique first sentences
   fell 1.000 -> 0.900 and max duplicates rose 1.00 -> 1.60.

3. **Quality-only RL learns "write longer" (+30.8 words); the diversity reward
   removes that incentive entirely (+0.7 words).** Spreading out apparently pays
   better than padding. It also means E1's diversity gain cannot be a length
   artifact.

4. **E0 and E1 travel the SAME distance in weight space** (LoRA update norms
   within 5% at every matched checkpoint) despite E1 running 1.5-1.8x the KL.
   The diversity term **redirects** an equally-sized update rather than adding
   movement. Both arms put ~3x more update mass in `gate_proj`/`up_proj` than in
   `k_proj`/`v_proj` — the MLP/phrasing path, not the attention/selection path.
   Plausible mechanistic reason diversity is hard to move.

5. **The effect needed ~150 steps to emerge from noise.** At batch 88 E1 was
   statistically indistinguishable from E0 on diversity and I nearly concluded
   alpha=0.5 was too weak. Anyone running 100 steps would have concluded
   diversity rewards don't work.

6. **Length explains only 1.4-2.3% of between-group diversity variance**, so the
   length confound is real but negligible. Measured, not assumed.

---

## 6. Things I got wrong mid-flight (so you don't trust stale claims)

- Called "E0 is demonstrating mode collapse" at 50 batches. **Retracted** — it
  was noise; the signs flipped by batch 77.
- Claimed the KL curve's rise-then-settle was "the KL penalty pulling the policy
  back". **Wrong** — the penalty is 0.4% of the loss and cannot do that.
- Reported E1's entropy as *rising* +3.0% from a mid-run window. **Over the full
  run it FELL -3.1%**, slightly more than E0's -2.1%. The "arms move in opposite
  directions on entropy" story does **not** survive. What separates the arms is
  semantic diversity, not token entropy.
- Presented a 98.7th-percentile prompt (*Aethel* x12) as typical collapse.

**Rule adopted:** no trend claims until an arm completes and the first-25% vs
last-25% comparison clears 2*SE with 75 batches per side.

---

## 7. Open recommendations for the next iteration

1. **Set beta (KL) to 0.** Measured at 0.4% of loss magnitude, so it's already
   near-inert — but the principled argument is stronger: the reference model
   *is* the collapsed distribution (eff. rank 2.0/16), so a KL penalty
   regularizes **toward** the pathology under study. The programmatic gates do
   the job KL is usually there for, without that conflict of interest.
   (Not changed mid-study because beta must match across arms.)
2. **Raise alpha.** E1 at alpha=0.5 gained +0.0200 deviation. The frontier is
   nearly flat (corr(quality, eff_rank) = -0.108 across prompts), so there is
   slack to spend. Try alpha = 1.0-2.0.
3. **Add unique-opening-rate to the reward**, not just to eval. It caught what
   log-det missed.
4. **Consider more steps.** Both arms were still moving at 300.
5. **tau=5.0 barely binds** — 96.2% of stories clear it. It's a floor against
   gaming, not a selector. If you want it to select, use ~7.0 (the median).

---

## 8. How to resume

```bash
cd /workspace/creative-writing
source env.sh                    # keys, vLLM Blackwell workarounds, PYTHONPATH
./run_rest.sh                    # idempotent; skips finished stages via outputs/.stages/*.done
./run_8b.sh                      # 8B scaling arm (DPO only), run AFTER run_rest.sh
```

**Stage markers** live in `outputs/.stages/`. Delete a marker to force a re-run.

**Per-arm study** (`between_arms.sh <ARM> <VERIFIED_PID>`) stops the
orchestrator, runs `ckpt_study.py` on the freed GPU, then relaunches. **Always
verify the PID with `ps -p $PID -o cmd=` before passing it** (see bug 2.8).

### Key files
| path | what |
|---|---|
| `src/diversity.py` | deviation, log-det, leave-one-out marginals, effective rank, greedy subset |
| `src/gates.py` | completeness, length window, 4-gram loops, entropy floor, non-ASCII |
| `src/judge.py` | per-story absolute rubric, sqlite cache, health/assert_healthy |
| `src/rewards.py` | GDPO channels, quality-conditioned diversity credit |
| `src/train_grpo.py` / `train_dpo.py` | the five arms |
| `src/ckpt_study.py` | generate + read stories across every checkpoint |
| `src/qualitative.py` | collapse-signature profiler (openings, tics, register spread) |
| `src/evaluate.py` / `make_report.py` | eval harness + frontier report |
| `logs/experiments/*.md` | one report per experiment |
| `NOTES.md` | this file |

### Tests — 56, all passing
```bash
EMB_DEVICE=cpu CUDA_VISIBLE_DEVICES= python3 src/test_diversity.py     # 15
EMB_DEVICE=cpu CUDA_VISIBLE_DEVICES= python3 src/test_rewards.py       # 10
EMB_DEVICE=cpu CUDA_VISIBLE_DEVICES= python3 src/test_pairs.py         # 14
EMB_DEVICE=cpu CUDA_VISIBLE_DEVICES= python3 src/test_eval_metrics.py  # 17
```

---

## 9. Housekeeping / TODO

- [ ] **ROTATE THE OPENROUTER KEY.** `prior_run/train.py:30` has a live key in
      plaintext. The HF repo is public — the published copy is redacted, but the
      key itself is live. Judge spend so far ~$14.2 of $30.
- [ ] **Finish the HF upload.** `Mercity/creative-writing-llm` is public,
      144 files / 2.43 GB. ~790 MB of E1 checkpoints + everything E1 produced
      after step 300 still outstanding. Upload commands were blocked by the
      permission classifier; needs approval or a manual run:
      `source env.sh && unset HF_HUB_ENABLE_HF_TRANSFER && python3 hf_upload/upload.py`
      (idempotent, resumable via `hf_upload/.done/`).
- [ ] Model card says E1 stops at step 200/300 — stale, needs refresh.
- [ ] `logs/experiments/04_E0_checkpoint_story_study.md` describes the **earlier
      partial 163-step E0 run at lr 3e-6**; `outputs/ckpt_study/E0-baseline/metrics.csv`
      is from the final 300-step run. **Trust the CSV.** Report needs updating.
- [ ] Nothing has been deleted since the user asked — except ~4 GB of
      `optimizer.pt` removed from E0/E1 checkpoints BEFORE that instruction.
      All adapters intact; only resume-from-checkpoint capability was lost.

## 10. Cost / resource ledger

- Judge (OpenRouter): **~$14.2 of $30**. ~$4 projected for the remaining arms.
- Disk: 37 GB free of 70 GB. Checkpoints are 253 MB (adapter only) or 768 MB
  (with optimizer state — E2 onward preserves it).
- VRAM: stable ~29.3 GB of 32.6 GB during GRPO. No OOM since the micro-batch
  went to 1.
- Wall clock: ~1h50m per 300-step GRPO arm; ~5 min per DPO arm; ~8 min per
  checkpoint story study; ~45 min for the 16k-story pool.

---

## 11. Late-session findings (added ~02:15 IST)

### 11.1 E0 and E1 differ ONLY in direction, not magnitude or location
At final checkpoint (step 300), LoRA update magnitude ||B@A||*(alpha/r):

```
total:         E0 3.4431   E1 3.3854   ratio 0.983
per-module:    all seven projections, ratio 0.972-0.993
depth blocks:  all six blocks of 36 layers, ratio 0.957-1.013
```

**E1 gets 5-6x the diversity gain with a 1.7% SMALLER weight update, distributed
identically across modules and depth.** The diversity reward does not find a
different part of the model to change — it finds a different *direction* within
the same subspace. Corollary: "diversity is expensive to optimize" is false
here; it costs no extra parameter movement.

Both arms put ~3x more update mass in `gate_proj`/`up_proj` (MLP / phrasing)
than in `k_proj`/`v_proj` (attention / selection). Plausible reason diversity is
hard to move: the optimizer is rewording sentences, not choosing different
narratives.

### 11.2 Two temperatures now on disk
- `outputs/ckpt_study/<arm>_T0.9/` — original study, T=0.9 top_p=0.95, seed 20260816
- `outputs/ckpt_study/<arm>/`      — T=1.0 top_p=1.0, seed 4242 (unfiltered sampling)

T=1.0 removes top-p truncation, so absolute values are NOT comparable across the
two; the E0-vs-E1 CONTRAST is. If E1 beats E0 at both temperatures the result is
not a sampling artifact. **Nothing was overwritten — T=0.9 was copied first.**

### 11.3 E2 was stopped deliberately
Killed at step ~4/300 at the user's request ("we don't need E2"); its partial
output dir was removed. `configs/E2_div_group.yaml` is ready and correct
(lr 3e-5, save_only_model false so it preserves optimizer state).
To run it: `./run_rest.sh` — it will skip E0/E1 via the stage markers.

### 11.4 Falsifiable prediction still to check
**n-gram metrics (distinct-4, self-BLEU) should separate E0 and E1 far LESS than
effective rank does**, because the collapse is tonal/structural, not lexical.
The eval harness (base/E0/E1, 30 held-out prompts x 16 samples) was queued at
the end of the session and writes `outputs/eval/results.csv`. If it completed,
check that file first thing — it is the cleanest test of whether the embedding
metrics are measuring the right thing.

### 11.5 CORRECTION — "verbatim opening collapse" was largely a top-p artifact

Ran the checkpoint study at BOTH T=0.9/top_p=0.95 and T=1.0/top_p=1.0:

```
                     base uniq   step300 uniq   base maxdup   step300 maxdup
E0  T=0.9 top_p.95      1.000        0.900          1.00          1.60
E0  T=1.0 top_p1.0      0.950        0.983          1.30          1.10
E1  T=0.9 top_p.95      1.000        0.883          1.00          1.70
E1  T=1.0 top_p1.0      0.950        0.967          1.30          1.20
```

**The direction of the effect FLIPS with sampling settings.** Under top-p 0.95,
opening uniqueness falls with training; unfiltered, it rises. The dramatic
"5 of 6 stories open with 'The city didn't sleep.'" result at step 300 does NOT
reproduce at T=1.0 — the same checkpoint gives six distinct openings.

**Mechanism:** top-p truncation concentrates probability on the single most
likely opening; training sharpens that peak; the truncation then snaps every
sample onto it. It is an interaction between training and the decoding
truncation, not a property of the policy distribution.

**Consequences:**
- Retract the recommendation to add unique-opening-rate to the REWARD. It would
  have optimized against a decoding artifact.
- Any claim about positional/opening collapse must state its sampling settings.
- **E1's advantage on effective rank and deviation appears at BOTH temperatures**,
  so that result is NOT a sampling artifact. This is the main reason running both
  was worth it.
- General lesson: before believing a qualitative collapse finding, re-check it
  under different decoding settings.

### 11.6 Per-prompt breakdown — the gain is CONCENTRATED, and the 10-prompt study is underpowered

```
                    E1 wins   mean D    top-2 prompts' share of total gain
T=1.0 (10 prompts)    7/10    +0.035              65%
T=0.9 (10 prompts)    8/10    +0.089              74%
```

Sign test: 7/10 -> p ~ 0.17; 8/10 -> p ~ 0.055. **Neither is strong.** E1 loses on
2-3 prompts at each temperature, and the losing prompts DIFFER between
temperatures (eval-0002/0005 at T=1.0; eval-0004/0007 at T=0.9), which suggests
those flips are noise rather than systematic weaknesses.

**The headline rests on the TRAINING-RUN statistics (n = 4800 stories/arm,
deviation +0.0200 vs +0.0037), not on this study.** The 10-prompt checkpoint
study is directionally consistent at both temperatures = corroboration, not
proof. For a properly powered held-out test see `outputs/eval/results.csv`
(30 prompts x 16 samples per model).

**Do not quote the 10-prompt study as the primary evidence.**

### 11.7 HEADLINE FINDING — LLM-judge "novelty" does NOT measure diversity (n=15,870)

```
corr(judge novelty, story's own deviation from its peers) = +0.057    ~ZERO
corr(judge novelty, judge quality)                        = +0.799    novelty IS quality
corr(mean group novelty, group effective rank)            = +0.088
corr(mean group quality, group effective rank)            = +0.003
```

**An LLM judge's "novelty" score is ~80% a restatement of quality and carries
essentially no information about actual semantic distinctiveness.** Measured on
15,870 scored stories, so this is not a noisy estimate.

Generalizes beyond this project: **any diversity reward built on asking an LLM
"how novel is this?" is measuring craft, not variety.** This is exactly what the
prior run optimized, and why it produced surface variation with no real spread.
It is also why this study takes diversity from embeddings, not the judge.

Corroborating anecdote: in E0, judge novelty rose +0.519 (nearly double the
quality gain of +0.303) while embedding diversity stayed flat.

### 11.8 What E1 actually ADDED to the writing (T=1.0, forms in >=2 of 6 samples)

```
form                base   E0@300   E1@300   E1-E0
second person       0.00     0.10     0.50    +0.40
present tense       0.20     0.10     0.40    +0.30
dialogue-heavy      0.30     0.20     0.30    +0.10
comic/absurd        0.20     0.10     0.20    +0.10
solemn/elegiac      1.00     1.00     1.00     0.00
TOTAL forms         1.70     1.50     2.40    +0.90
```

- **E1 invented second-person narration**: base uses it on 0/10 prompts, E1 on 5/10.
- **E0 LOSES forms** (1.70 -> 1.50): quality-only training narrows the repertoire.
- **CEILING: solemn/elegiac is 1.00 in every condition.** Neither arm broke the
  tonal monoculture. E1 diversifies grammatical person / tense / mode while every
  story stays in the same melancholy register.

**Honest framing: the deviation reward broadens narrative FORM, not TONE.**
Next target should be tone/register specifically — e.g. a register-classifier
reward, or prompt-conditioned tone targets.

### 11.9 Inconclusive: is the gain larger where the base was more collapsed?
corr(base eff_rank, E1-E0 gain) = +0.163 at T=1.0 but -0.325 at T=0.9.
Opposite signs across temperatures with n=10. **No reliable answer; do not claim one.**

### 11.10 CONFIRMED — n-gram metrics are blind to this collapse
Base model, 480 stories over 30 held-out prompts (`outputs/eval/rows/base.json`):
```
distinct4   0.938      <- n-grams say "highly diverse"
self_bleu   0.287      <- n-grams say "highly diverse"
eff_rank    1.891/16   <- embeddings say "collapsed"
n_clusters  1.033      <- effectively ONE mode
tok_entropy 0.893
```
**The same 480 stories are rated diverse by surface metrics and collapsed by
embedding metrics.** This was predicted in advance from reading the prose (§1)
and is now confirmed. Anyone evaluating creative diversity with distinct-n or
self-BLEU alone would conclude this model is fine.

### 11.11 PREDICTION — DivPO-prob will underperform DivPO-emb
Pair statistics BEFORE either arm was trained:
```
             chosen dev  rejected dev   MARGIN    chosen logp  rejected logp
divpo_emb      0.1595      0.1151      +0.0444      -1.228       -1.208
divpo_prob     0.1341      0.1309      +0.0031      -1.372       -1.086
multipos       0.1432      0.1246      +0.0186
```
`divpo_prob` separates pairs sharply on **model probability** (correct direction)
but its pairs are **essentially identical in semantic deviation** — a 14x weaker
diversity signal than `divpo_emb`.

**Low model probability != semantically distinct.** "Reject the near-greedy
sample" selects for surprising WORDING, not a different STORY — the same failure
mode as judge-novelty via a different route.

**Falsifiable prediction: E4b (divpo-prob) will show little/no diversity gain
over E4a (divpo-emb).** Check this first when the DPO arms run.

E3 DDPO loss weights are live and well spread: mean 1.000, sd 0.298, range
0.333-2.665.

### 11.12 DEFINITIVE HELD-OUT RESULT (`outputs/eval/results.csv`)
480 stories/model, 30 held-out prompts x 16 samples, T=0.9, identical seed.

| model | quality | eff_rank | pairwise | logdet | distinct4 | self_bleu | ends_cleanly |
|---|---|---|---|---|---|---|---|
| base | 6.472 | 1.891 | 0.1179 | -32.46 | 0.9378 | 0.2875 | 0.975 |
| E0 quality-only | 6.716 | 1.957 | 0.1259 | -31.59 | 0.9485 | 0.2635 | 0.994 |
| **E1 +deviation** | **6.864** | **2.081** | **0.1390** | **-30.08** | 0.9464 | 0.2625 | 0.981 |

Gains over base:
```
E0:  eff_rank +0.066  dev +0.0080  logdet +0.873  quality +0.244
E1:  eff_rank +0.190  dev +0.0211  logdet +2.380  quality +0.392
     E1/E0:     2.9x       2.6x         2.7x          1.6x
```
**E1 wins on BOTH axes — 2.9x the diversity gain AND 1.6x the quality gain.**
Not a tradeoff. Judge health on this eval: 496 calls, 1 failure (0.2%).

**n-gram metrics fail to separate the arms at all:**
```
              eff_rank        distinct4        self_bleu
E0 vs base    +0.066          +0.0107          -0.0240
E1 vs base    +0.190 (2.9x)   +0.0085          -0.0250   <- ~identical
```
distinct-4 actually rates E0 HIGHER than E1. **Evaluating creative diversity
with n-gram metrics would have concluded these two models are the same.**
This is the methodological headline of the study.

### 11.13 Qualitative read (full report: logs/experiments/05_qualitative_read.md)

**E0 is a better writer telling the same story. E1 tells different stories.**

Template reuse — fraction of step-300 samples whose first 8 words verbatim-match
a BASE opening for that prompt:
```
E0 (quality-only): 19/60 = 31.7%
E1 (+deviation):    7/60 = 11.7%    2.7x less reuse
```
Tracks the 2.9x effective-rank separation on held-out eval.

- **E0 preserves the base frame and polishes inside it.** Openings often
  near-verbatim to base; improvements are internal (one graduation sample
  develops a "Year One/Two/Three" structure the base never attempts, with far
  more specific detail). Exactly what a per-story quality judge rewards.
- **E1 changes entry point, premise and POV.** Graduation: base+E0 always open
  AT the podium; E1 opens in retrospection. Martial arts: base+E0 write a humble
  student; E1 writes an arrogant one in a hoodie reading "I Know Everything" —
  confrontation instead of communion. Ash/snow: base+E0 use one template (named
  lone adult, rural dwelling, remembering); E1 shifts to children/collective/city
  scale. Black Friday: base+E0 both use "The air in the X Mall…"; E1 uses none.
- **CEILING: solemn/elegiac = 1.00 in EVERY condition.** E1 diversifies person,
  tense, scale, POV and premise but NOT tone. Nobody wrote a comedy, even for an
  explicitly comic prompt. **Next objective to target is register explicitly.**

### 11.14 CORRECTION to 11.8 — "nobody wrote a comedy" was too strong
After reading the Cthulhu and NYC stories IN FULL (not just openings):

**E1 does reach comic/absurd register.** Examples the base model never
approaches: Cthulhu waking on a Manhattan balcony and making a radio tower play
Chopin's Nocturne, then watching a dog chase a ball; and "a concert in Helsinki
where a hundred thousand people played accordions in perfect unison, each note
tuned to a specific frequency of sea bass in the Barents Sea."

The `comic/absurd` keyword regex scored these as non-comic because the humour is
**situational, not lexical**. So the 1.00/1.00/1.00 solemn-register table
UNDERCOUNTS E1's tonal range. The monoculture ceiling is real but softer than
stated. **Lesson: keyword-based register detection misses situational humour —
do not trust it as the sole tonal metric.**

**E1 also fulfils the prompt better in at least one case.** The NYC prompt asks
"Why?" — base and E0 give atmospheric vignettes with no mechanism ("No one knew
why. No one asked."). One E1 sample writes a dialogue-driven SF scene with an
actual causal device ("I stopped the *intent*"), the only sample across all
three conditions that answers the question. Another E1 sample writes in present
tense and refuses the consoling ending (violence returns at midnight), where
base and E0 both resolve into calm.