[missing figure: {p.name}]
" b = base64.b64encode(p.read_bytes()).decode() return f"
"
def tbl(rows, cols=None, hi=None) -> str:
if not rows:
return "(no data)
" cols = cols or list(rows[0].keys()) h = "".join(f"The base model is already collapsed before any RL. Across a 16,000-story pool (1,000 prompts × 16 samples), effective rank is 2.006 out of a ceiling of 16 — sixteen stories for one prompt span roughly two effective semantic directions, at ~0.87 mean cosine similarity. This reframes the study: the question is not whether RL causes collapse, but whether any objective can lift diversity off a floor that pretraining already imposed.
{tbl(poolrows, ["metric", "value"])} {img(FIGS / "03_pool_4b_baseline.png")}Identical data, seed, learning rate (3e-5), step count and LoRA config. 4,800 stories scored per arm. First quarter vs last quarter of each run.
{tbl(rows, ["metric", "E0 start", "E0 end", "E0 Δ", "E1 start", "E1 end", "E1 Δ", "E1/E0"], hi={"E1 Δ", "E1/E0"})}10 held-out prompts × 6 samples at T=0.9, fixed seed, generated from every checkpoint. 960 stories read across both arms.
{tbl(ckrows, ["arm", "step", "quality", "eff_rank (of 6)", "deviation", "logdet", "words"])} {img(FIGS / "E0-baseline_trajectory.png")}Opening sentences of all 6 samples per prompt. Base model shown once (identical starting point for both arms), then each arm at step 300.
{ex_html}480 stories per model: 30 held-out prompts x 16 samples, T=0.9, top_p=0.95, identical seed. Judge health on this run: 496 calls, 1 failure (0.2%).
{tbl(evrows, ["model","quality","eff_rank (of 16)","pairwise","logdet","ends cleanly","words"], hi={"eff_rank (of 16)"})}The same 480 stories per model, scored two ways:
{tbl(blind, ["model","Δ eff_rank (embed)","Δ logdet (embed)","Δ distinct-4 (n-gram)","Δ self-BLEU (n-gram)","Δ quality"], hi={"Δ eff_rank (embed)"})}10 held-out prompts x 6 samples from every checkpoint of both arms, at two sampling settings. Stories read in full for three prompts; openings and premises scanned for all ten.
E0's step-300 openings are frequently near-verbatim to base:
{tbl(qrows, ["prompt","base","E0 @ 300"])}The improvement is real but internal. E0's bodies are richer and better organised — one graduation sample develops an explicit “Year One / Year Two / Year Three” structure the base never attempts, with far more specific detail (“Jenna's red scarf”, “Jake's habit of drawing tiny suns on his H.W. papers”). That is exactly what a per-story quality judge rewards, and why E0's judge score rises +0.24 while its diversity does not move. E0 is a better writer telling the same story.
Graduation prompt. Base and E0 open at the podium, in the ceremony, in every sample. Two of E1's three open in retrospection instead — no stage, no lights, no crowd:
Martial-arts prompt. Base and E0 write the student as a humble supplicant (“I… I just want to learn”; “No app on her wrist. No headset. Just folded hands”). E1 rewrites the relationship into a confrontation:
Another E1 sample relocates the scene from dojo to neon city street; a third inverts the premise entirely — the student says “I didn't download anything. I just… felt it.”
“The ash turned to snow.” Base and E0 use one template in all six samples: a named lone adult, at a rural dwelling, remembering (Magda/cottage, Elena/clearing, Masahiro/temple; Marlow/garden, Eli/watchtower, Lyra/cottage). E1 breaks both scale and POV:
Black Friday prompt — the clearest case. Base uses “The air in the X Mall…” or “The sky burned crimson…” in all four sampled openings; E0 preserves it (“The air in the Glendale Mall…”, “The air in the Orchard Mall…”). E1 uses none of it: “No one remembers the date. The clocks stopped on a Tuesday.” / “The temperature dropped the second the lights went out.”
The NYC prompt asks “Why?” — it demands a mechanism. Base and E0 mostly supply atmospheric vignettes with no explanation (“No one knew why. No one asked.”). One E1 sample instead writes a dialogue-driven science-fiction scene that actually answers it — the only sample across all three conditions to supply a causal mechanism:
Another E1 sample writes in the present tense and refuses the consoling ending — the violence returns at midnight (“A man in a grocery bag gets his arm slashed by a scrawny boy, screaming”), where base and E0 both resolve into calm.
E1 does not fix verbatim opening duplication. At step 300, unique-opening rate is 0.883 for E1 vs 0.900 for E0 — marginally worse — and both arms have 1/10 prompts with ≥3 identical openings. What E1 gains is register spread (2.00→2.40 distinct forms, while E0 falls 2.00→1.40). The diversity reward broadens what kind of thing the model writes without fixing how it starts sentences.
Whole-story embeddings can miss positional collapse. In E0 one prompt went from 6 distinct openings to 5-of-6 identical while effective rank and deviation both drifted up. Unique-opening rate should be a first-class metric, not a diagnostic afterthought.
Effect sizes are modest in absolute terms — E1's held-out effective rank is 1.80 against a ceiling of 6. The floor was lifted, not escaped.
The effect needed ~150 steps to emerge from noise. At batch 88 E1 was statistically indistinguishable from E0. A 100-step study would have concluded diversity rewards do not work.
Entropy did not separate the arms. Both fell (E0 −2.1%, E1 −3.1%). An earlier mid-run window suggested E1's entropy was rising; that did not survive the full run. Token entropy and semantic diversity are dissociated — which is the point, but not in the direction first reported.