README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: peft
pipeline_tag: text-generation
tags:
- lora
- peft
- trl
- grpo
- gdpo
- dpo
- divpo
- rlhf
- diversity
- creative-writing
- mode-collapse
---

# Diversity-aware post-training for creative story generation (Qwen3-4B)

Research artifacts for a controlled study of **whether RL post-training can raise the
*semantic* diversity of short-story generation without paying for it in quality**.

Everything here is measured, not asserted: the repo carries the LoRA adapters, the
scored generation pool they were selected from, the per-step reward telemetry, the
judge-calibration evidence, the experiment reports, and the source that produced all
of it.

> ⚠️ **Work in progress.** Only **E0** (quality-only baseline) has finished its full
> 300-step run. **E1** was still training when this snapshot was uploaded — its
> checkpoints stop at **step 200 of 300**. **E2, E3 and E4 have not been trained yet**;
> their configs are present, their adapters are not. No cross-arm comparison exists yet,
> and none should be read into this repo.

---

## The finding this study is built around

Sampling the **base** policy 16 times per prompt over 1,000 prompts (16,000 scored
stories, `outputs/pool_4b/`) gives a mean **effective rank of 2.006 out of a ceiling of
16**. Sixteen independent samples of the same prompt span roughly two effective
directions in embedding space; mean pairwise distance is 0.132, i.e. same-prompt stories
sit at about 0.87 cosine similarity.

Quality-only RL does not fix this. E0 ran 300 steps of GRPO on the judge's quality score
alone and moved judge quality **+0.30** (6.557 → 6.860, first vs. last quarter of the run)
while the diversity statistics stayed flat (mean pairwise deviation 0.1354 → 0.1391;
mean group log-det −12.81 → −12.64). The held-out checkpoint study
(`outputs/ckpt_study/`, 10 prompts × 6 samples per checkpoint) tells the same story:

| step | judge quality | effective rank | deviation |
|---|---|---|---|
| 0 (base) | 6.65 | 1.677 | 0.130 |
| 100 | 6.78 | 1.655 | 0.128 |
| 200 | 6.75 | 1.700 | 0.135 |
| 300 | 7.03 | 1.712 | 0.139 |

Craft improves; the model still writes the same story six times. That gap is what the
diversity arms exist to close.

---

## Method

**Base policy.** `Qwen/Qwen3-4B-Instruct-2507`, LoRA **r=32, alpha=64, dropout 0.0** on all
attention and MLP projections (`q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj`).

**Online arms.** GRPO via **TRL 1.10** with `multi_objective_aggregation="normalize_then_sum"`,
which is a line-for-line implementation of **GDPO** (arXiv 2601.05242): each reward channel is
normalized *within its prompt group* before the channels are summed and the advantage is
normalized batch-wise. Generation is **vLLM colocated** in the training process. G=8 samples per
prompt, generation batch 16 (2 prompts × 8), `max_completion_length=1024`, lr 3e-5 with
`constant_with_warmup`, beta (KL) 0.02, 300 steps. Everything ran on a **single RTX 5090 (32 GB)** —
the 32 GB budget is what forced the 4B policy, since colocated GRPO needs two resident copies of
the weights.

**Offline arms.** DPO over preference pairs mined from the same 16k pool
(beta 0.1, sigmoid loss, lr 5e-6, epochs chosen for optimizer-step parity across arms).

**Reward channels.**

- `quality` — judge score, per story.
- `deviation` `d_i` — mean embedding distance from a sample to the rest of its group
  (`BAAI/bge-base-en-v1.5`), credited only if judge quality ≥ tau=5.
- `marginal` `m_i` — the sample's marginal contribution to the group's log-determinant
  (a set-level diversity volume), z-scored within group.

Every diversity credit is **per-sample by construction**. A single set-level scalar shared by a
whole group has zero within-group variance and therefore contributes exactly nothing to a GRPO
advantage — the failure mode diagnosed in `prior_run/` and written up in
`logs/experiments/00_prior_run_autopsy.md`.

**Judge.** `deepseek-v4-flash-0731` via OpenRouter, scoring **one story at a time on an absolute
rubric** (not batched relative scoring), with reasoning disabled. Calibrated before any training
against four cells with known ordering: detects incoherence (Δ 2.25), detects repetition (Δ 4.67),
not saturated (sd 0.76). See `logs/experiments/01_judge_calibration.md`.

**Programmatic gates.** Completeness, 150–600 word window, 4-gram / line loop detection and
`finish_reason == "length"` are checked deterministically before a story is ever judged, so
truncated or degenerate text cannot earn reward through any channel.

### The arms

| arm | type | reward / objective | status in this repo |
|---|---|---|---|
| **E0-baseline** | GRPO | quality only — the collapse control | ✅ complete, 300 steps + `final` |
| **E1-div-individual** | GRPO | quality + pairwise deviation `d_i` (alpha 0.5, gated at tau=5) | 🟡 partial, steps 50–200 of 300 |
| **E2-div-group** | GRPO | quality + deviation + log-det marginal `m_i` (alpha 0.5, gamma 0.5) | ⬜ config only, not trained |
| **E3-multipos** | DPO | multi-positive, deviation-weighted loss; 4 greedy-diverse chosens vs. rotating negatives (3,593 rows) | ⬜ config + pairs only |
| **E4a-divpo-emb** | DPO | faithful DivPO, embedding-deviation criterion (956 rows, rho=7) | ⬜ config + pairs only |
| **E4b-divpo-prob** | DPO | faithful DivPO, lowest length-normalized logprob among quality ≥ rho | ⬜ config + pairs only |

---

## Repository layout

```
outputs/
  E0-baseline/            LoRA adapters: checkpoint-{50..300} + final
                          reward_history.json     per-step reward telemetry (300 steps)
                          trl_log_history.json    full TRL log history
                          judge_cost.json         judge call/token/USD accounting
  E1-div-individual/      same structure, checkpoints 50-200 (run in progress)
  pool_4b/                pool_train.jsonl        16,000 scored base-policy stories
                          emb_train.npy           their bge-base-en-v1.5 embeddings
                          summary_train.json      pool-level summary stats
  pairs_4b/               divpo_emb_train.jsonl, divpo_prob_train.jsonl,
                          multipos_train.jsonl, pair_stats_train.json
  ckpt_study/E0-baseline/ stories.md, raw.json, metrics.csv  (per-checkpoint story dumps)
logs/
  experiments/            the written reports (start here)
  figures/                plots referenced by the reports
configs/                  one YAML per arm, with the reasoning for every knob in comments
src/                      all Python: training, rewards, diversity, judge, gates, eval, tests
data/                     train/eval prompt splits + split metadata
prior_run/                the previous, failed run's artifacts (see autopsy)
```

**Read the reports in this order:** `00_prior_run_autopsy.md` → `01_judge_calibration.md` →
`02_setup_and_deviations.md` → `03_pool_4b_baseline.md` → `04_E0_checkpoint_story_study.md` →
`E0-baseline.md`.

### Data schemas

`outputs/pool_4b/pool_train.jsonl` — one story per line:
`prompt_id, prompt, idx, text, n_words, n_tokens, finish_reason, gate_passed, gate_reasons,
ends_cleanly, quality, novelty, deviation, marginal, group_logdet, mean_logprob, cumlogprob`.
Row *i* of `emb_train.npy` corresponds to line *i* of the JSONL.

`outputs/pairs_4b/*.jsonl` — DPO rows: `prompt_id, prompt, chosen, rejected` (+ per-row weight
for the multi-positive set).

`data/` — 1,000 train and 50 eval prompts (seed 42), filtered to 10–60 words from 272,600 scanned;
`split_meta.json` also carries the exact system prompt used for every generation in the study.

---

## Using an adapter

Each checkpoint directory is a standard PEFT adapter (`adapter_model.safetensors` +
`adapter_config.json`, ~253 MB each).

```python
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

path = snapshot_download(
    "Mercity/creative-writing-llm",
    allow_patterns="outputs/E0-baseline/final/*",
)

tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B-Instruct-2507")
base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B-Instruct-2507", dtype="auto", device_map="auto"
)
model = PeftModel.from_pretrained(base, f"{path}/outputs/E0-baseline/final")

system = (
    "You are a fiction writer. Write a complete short story of 200-500 words responding "
    "to the writing prompt.\nWrite only the story: no title, no preamble, no commentary, "
    "no author's note.\nFinish inside the word budget. The story must reach a real ending, "
    "not stop mid-scene."
)
msgs = [{"role": "system", "content": system},
        {"role": "user", "content": "A lighthouse keeper receives a letter addressed to the sea."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=1024, temperature=1.0, top_p=1.0, do_sample=True)
print(tok.decode(out[0][ids.shape[-1]:], skip_special_tokens=True))
```

Swap `outputs/E0-baseline/final` for `outputs/E0-baseline/checkpoint-150`,
`outputs/E1-div-individual/checkpoint-200`, etc. The study sampled at T=1.0 during training and
T=0.9 / top_p=0.95 for the checkpoint story study.

---

## Selected numbers

**Base-policy pool** (`logs/experiments/03_pool_4b_baseline.md`, n=16,000):

| metric | value |
|---|---|
| gate pass rate | 0.992 |
| judge quality (mean ± sd) | 6.447 ± 0.833 |
| mean pairwise deviation | 0.1321 |
| mean group log-det | −30.42 |
| **mean effective rank (ceiling 16)** | **2.006 ± 0.323** |
| corr(quality, effective rank) | −0.108 |
| corr(deviation, effective rank) | 0.992 |

Quality and diversity are **largely independent** across prompts (r = −0.108), so a method that
raises diversity without lowering quality is exploiting existing slack rather than defying a
tradeoff.

**E0 run** (`logs/experiments/E0-baseline.md`, 300 steps, 4,800 stories judged, verdict *healthy*):

| metric | early (first 25%) | late (last 25%) | Δ |
|---|---|---|---|
| judge quality (passing) | 6.557 | 6.860 | **+0.303** |
| mean deviation | 0.1354 | 0.1391 | +0.004 |
| mean log-det | −12.81 | −12.64 | +0.165 |
| gate pass | 0.9925 | 0.9875 | −0.005 |
| policy entropy | 1.2535 | 1.2274 | −0.026 (−2.1%) |

No entropy collapse, and the reward-hacking trip conditions (diversity up while quality or
validity falls) were not tripped. Judge cost for the whole E0 run: **$0.84** over 5,109 calls.

**Qualitative** (`logs/experiments/04_E0_checkpoint_story_study.md`): the collapse is a *tonal
monoculture*, not a lexical one — 92–95% of stories carry solemn/elegiac vocabulary against ~15%
comic, even on explicitly comic prompts, and verbatim opening duplication *rises* with
quality-only training. This predicts that n-gram metrics (distinct-4, self-BLEU) will separate the
arms far less than effective rank does.

---

## Environment

torch 2.13.0+cu130 · vLLM 0.27.1 · transformers 5.15.0 · TRL 1.10.0 · PEFT 0.20.0 ·
sentence-transformers 5.7.0 (`BAAI/bge-base-en-v1.5`) · single RTX 5090, 32.6 GB, sm_120.
FlashInfer is disabled (it misdetects sm_120); Liger is disabled because it suppresses TRL's
`entropy` metric.

## Caveats and honest notes

- **E1 is mid-flight.** Its checkpoints are a snapshot at step 200/300 and its reports do not
  exist yet. E2/E3/E4 are unrun. Do not read arm comparisons out of this repo.
- **`04_E0_checkpoint_story_study.md` documents an earlier, partial 163-step E0 run** (at the
  pre-correction lr 3e-6). `outputs/ckpt_study/E0-baseline/metrics.csv` is the *regenerated*
  version covering all checkpoints of the final 300-step run; where they disagree, trust the CSV.
- **Learning rate deviates from the plan** (3e-5, not 3e-6): at 3e-6 the adapter was effectively
  frozen — KL pinned near 8e-4 for 171 steps with every metric inside its noise band. The evidence
  is recorded in the config comments and `02_setup_and_deviations.md`.
- **Optimizer/scheduler/RNG state is not included** for E1's checkpoints. It is resume-only state,
  is rotated away by `save_total_limit`, and none of it is needed to load or evaluate an adapter.
- **`prior_run/train.py` is a redacted copy** — the original contained a hard-coded OpenRouter API
  key, replaced here with a placeholder. Nothing else in the file was changed.
- **Judge is not perfect.** It fails the truncation-sensitivity check (A−B = 1.33 against a 2.0
  bar), which is tolerated only because truncation is caught deterministically by the gates before
  any story reaches the judge. Pool-scoring judge failure rate was 0.45%.
- Every deviation from the original plan is enumerated in `logs/experiments/02_setup_and_deviations.md`
  rather than discovered in a footnote.

## References

- GDPO — group-wise reward normalization for multi-objective RL, arXiv 2601.05242 (Liu et al., NVIDIA)
- DivPO — Lanchantin et al., 2025 (diverse preference optimization)
- Effective rank — Roy & Vetterli, the exp-entropy of the Gram spectrum, used here as the primary
  continuous mode-count metric (k-means + silhouette was measured to be unable to separate collapse
  from spread and was demoted)

## License

Apache-2.0, matching the `Qwen/Qwen3-4B-Instruct-2507` base model. The prompt splits derive from a
public writing-prompts corpus; stories in `outputs/pool_4b/` are model generations.


## Status (updated — E0 and E1 both complete, 300 steps each)

| arm | status | result |
|---|---|---|
| **E0** quality-only GRPO | ✅ complete, 300 steps | quality +0.303, diversity flat |
| **E1** + pairwise deviation | ✅ complete, 300 steps | **diversity 5–6× E0's gain**, quality +0.243 |
| E2 div-grpo-group (log-det marginal) | ⏸ stopped at step 4, config ready | — |
| E3 multi-positive weighted DPO | ⏸ pairs built (3593 rows), not trained | — |
| E4a/E4b DivPO emb/prob | ⏸ pairs built (956 rows each), not trained | — |

### Held-out evaluation (30 prompts × 16 samples = 480 stories/model)

| model | quality | eff_rank (of 16) | pairwise | logdet | distinct-4 | self-BLEU |
|---|---|---|---|---|---|---|
| base | 6.472 | 1.891 | 0.1179 | −32.46 | 0.9378 | 0.2875 |
| E0 quality-only | 6.716 | 1.957 | 0.1259 | −31.59 | 0.9485 | 0.2635 |
| **E1 +deviation** | **6.864** | **2.081** | **0.1390** | **−30.08** | 0.9464 | 0.2625 |

**E1 wins on both axes:** vs base it gains 2.9× E0's effective-rank improvement
*and* 1.6× E0's quality improvement. Not a diversity-for-quality trade.

**Key methodological finding:** embedding metrics separate the arms by 2.9×;
n-gram metrics (distinct-4, self-BLEU) do not separate them at all — distinct-4
actually rates E0 *higher*. The collapse is tonal/structural, not lexical.

**Key finding on LLM judges:** across 15,870 scored stories,
`corr(judge "novelty", a story's actual embedding deviation) = +0.057`, while
`corr(judge "novelty", judge quality) = +0.799`. An LLM judge's novelty score is
~80% a restatement of quality and carries almost no information about semantic
distinctiveness.

See **`REPORT.pdf`** for the full write-up (metrics, figures, story examples,
qualitative read) and **`NOTES.md`** for the complete engineering log including
every bug and every retracted claim.