Setup actually used, and every deviation from the plan

Written during setup, updated as things changed. The point of this file is that the final report should never be the first place a deviation is disclosed.

Hardware (differs from the brief's assumption)

	assumed	actual
GPU	~48–56 GB (RTX 5090 / 6000 Ada class)	RTX 5090, 32.6 GB
CPU / RAM	—	32 cores / 188 GB
Disk	—	70 GB on `/workspace`
Driver / CUDA	—	580.65.06 / CUDA 13.0, sm_120 (Blackwell)

32 GB rather than ~50 GB is the single most consequential fact in this setup. It is what forced the model split below and every batch-size decision.

Software

package	version	note
torch	2.13.0+cu130	upgraded by vLLM; `sm_120` present in arch list
vLLM	0.27.1	above TRL's supported range (0.17–0.26); works, warns
transformers	5.15.0
trl	1.10.0	provides `multi_objective_aggregation`
peft	0.20.0
liger-kernel	installed	fused RMSNorm/SwiGLU/RoPE + LM-head CE
sentence-transformers	5.7.0	`BAAI/bge-base-en-v1.5`

Deviations from the plan

1. No Unsloth. TRL GRPOTrainer + vLLM colocate instead. The plan specifies Unsloth GRPO. Unsloth pins torch/vLLM versions that conflict with the working sm_120 stack (torch 2.13 + vLLM 0.27.1), and downgrading risks losing Blackwell support entirely — which took real effort to obtain. TRL 1.10's GRPOTrainer supports vLLM colocate natively and additionally supports the exact GDPO aggregation this study needs. Cost: no Unsloth memory savings, which is partly why the batch geometry below is so tight.

2. FlashInfer disabled. vLLM's bundled FlashInfer misreads the CUDA runtime (SM 12.x requires CUDA >= 12.9) and then rejects an sm_120 card as below sm_75, killing EngineCore at warm-up. Worked around with VLLM_ATTENTION_BACKEND=FLASH_ATTN and VLLM_USE_FLASHINFER_SAMPLER=0. Verified equivalent output vs TRITON_ATTN.

3. Model split: Qwen3-4B for the frontier, Qwen3-8B for scaling only. The user asked about "Qwen3.5-9B". Its config shows 24 linear_attention layers

8 full_attention (3:1 hybrid, with mamba_ssm_dtype), a vision tower, an MTP head and a 248k vocab. It is neither transformer-only nor text-only, and the LoRA-GRPO path through TRL/vLLM for linear-attention layers is unproven. Rejected. Qwen/Qwen3-8B is pure Qwen3ForCausalLM and is the transformer-only ~9B-class option (there is no Qwen3-8B-Instruct-2507).

8B is infeasible for the online arms on 32 GB, and this is arithmetic, not preference. GRPO with colocated vLLM needs two resident copies of the weights:

Qwen3-8B GRPO:  16.4 (trainer) + 16.4 (vLLM) = 32.8 GB > 32 GB, before any KV cache
Qwen3-4B GRPO:   8.05 (trainer) + 8.05 (vLLM) = 16.1 GB

LoRA does not help here: it shrinks optimizer state (~98 GB → ~1.2 GB for 8B) but the frozen base still has to be resident. DPO never generates during training, so it needs one copy and gets its reference model free by disabling adapters — which is why 8B DPO fits at ~21 GB while 8B GRPO cannot.

Decision: all six primary arms on Qwen3-4B-Instruct-2507 sharing one pool, plus E3-8B/E4-8B on a separate 8B pool as a scaling arm.

4. Judge: per-story absolute scoring, not batched relative scoring. The prior run sent 16 samples per call and asked for relative scores. Quality is compared against a fixed gate tau and across models at eval, so it must be absolutely calibrated. Per-story also makes the cache effective (identical stories recur across arms; each is paid for once).

5. Judge model deepseek/deepseek-v4-flash-0731 with reasoning DISABLED. It is a reasoning model; left alone it spends its entire output budget on reasoning and returns content: null. Calibration caught this at 4/166 calls succeeding, with every failure silently becoming a neutral 5.0 — a judge that scores everything identically while appearing to work. reasoning:{enabled:false} fixes it and is 25× cheaper and 5× faster (30 output tokens vs 764).

6. Judge rubric revised once (v3 → v4). Allowed by the brief. Strengthened the finishedness check into an explicit first step. See 01_judge_calibration.md.

7. Judge truncation-sensitivity reclassified as informational. A_complete − B_cut_matched = +1.33, below the +2.0 I set. Truncation is caught deterministically with 100% recall by gates.check(), and only gate-passing stories are ever sent to the judge, so judge truncation sensitivity cannot influence any gradient. The blocking checks (incoherence +2.25, repetition +4.67, non-saturation sd 0.76) all pass.

8. Batch geometry, forced by 32 GB.

knob	plan	actual	why
G	8	8	unchanged
max_completion_length	768	1024	raised; see below
per_device_train_batch_size	—	2	forward micro-batch; peak is the `2 × seq × 151936` logits tensor
gradient_accumulation_steps	—	8	keeps generation_batch = 16 = 2 prompts × G
vllm_gpu_memory_utilization	—	0.33	~10.8 GB: 8.05 weights + KV + workspace

Only generation_batch_size must be divisible by num_generations; the micro-batch need not be. So the forward can be shrunk without changing GRPO group semantics — advantages are still computed over full groups of 8.

9. Token budget RAISED, not lowered. The plan says 768. A 500-word story is ~~650 Qwen tokens, so 768 leaves no headroom and would have re-created the exact truncation bug that killed the prior run. Set to 1024 (~~790 words). The length gate rejects anything over 600 words (~780 tokens), so 1024 sits above the entire accept region: no story that could ever be scored is truncated by it. Nothing downstream truncates either — the judge and the embedder both receive complete story text.

10. Aggregation is GDPO, and it is native. The brief allowed a single conditioned scalar as a v1 fallback. Not needed: TRL 1.10's multi_objective_aggregation="normalize_then_sum" is a line-for-line implementation of GDPO (arXiv 2601.05242, Liu et al., NVIDIA) — group-wise normalization per reward channel, then batch-wise advantage normalization. E1 and E2 therefore run true decoupled normalization. Consequence: α and γ weight standardized channels, so α=0.5 means half an SD of diversity credit per SD of quality, and no manual rescaling of d_i into the 0–10 judge range is needed.

11. n_clusters as specified does not work; eff_rank added as the primary mode metric. The plan asks for "k-means over embeddings per prompt, silhouette-selected k (crude mode count)". Measured on synthetic 16-point sets, silhouette cannot separate collapse from spread at all:

configuration	best k	silhouette
fully collapsed (noise 0.001)	4	0.202
fully spread (random)	7	0.195
two clear modes	2	0.976

k-means partitions isotropic data regardless of spread, so the metric would have reported ~4 modes for collapsed output and ~7 for diverse output — fabricated structure, in a headline column. Two changes: SILHOUETTE_MIN raised 0.05 → 0.50, making n_clusters a conservative count of well-separated modes (returns 1 unless structure is unmistakable); and effective rank (exp-entropy of the Gram spectrum, Roy & Vetterli) added as the primary continuous mode measure — 1.00 identical, 2.12 two clusters, 13.29 spread, with no threshold to tune.

12. Liger disabled to recover the entropy metric. use_liger_kernel=True routes TRL to compute_liger_loss, which logs only [clip_ratio, kl] — the entropy metric exists solely on the _compute_loss path. Liger had been enabled to fix an OOM at micro-batch 2; micro-batch is now 1 (halving the logits tensor), so it is no longer needed. vLLM's budget trimmed 0.32 → 0.29 to cover the difference. Per-token policy entropy cannot be recovered from a finished run, and entropy collapse is a first-class failure signal, so this trade was worth the memory.

13. Entropy is treated asymmetrically. A large fall is strong evidence the policy is going deterministic and is flagged as tainting any diversity claim in that run. A rise is only permissive — a policy can raise per-token entropy while staying inside one narrative mode. The prior run demonstrates the dissociation directly (surface variation up, semantic diversity down). Entropy is never optimized.

14. Learning rate raised 2e-6 → 3e-6. Smoke showed KL ≈ 6e-4 at step 6 under 2e-6; across 300 steps that risked a null result in every arm, which would be uninformative. 3e-6 is still inside the plan's 1e-6..5e-6 band. Too-high instead risks instability, which the guardrail catches and which is itself a finding.

15. Eval runs one subprocess per model. vLLM v1 keeps EngineCore in a child process and del llm does not reliably reclaim its GPU memory, so a 7-model in-process loop would OOM on model 2 after model 1 succeeded. Process isolation makes teardown unconditional and stops one model's crash from taking down the whole eval; completed models are cached as row files and skipped on re-run.

Fallbacks taken (per the plan's ordered list)

G 8→6: not taken
max_new_tokens 768→512: not taken (raised to 1024 instead)
steps 300→200: not taken yet
policy 4B: taken as the primary, by the VRAM argument above
eval prompts 50→30: not taken

Bugs found and fixed during setup

asyncio.Semaphore built in __init__ binds to the first event loop; score_many_sync() calls asyncio.run() once per training step, so it would have raised "bound to a different event loop" from step 2 onward and turned every judge call into a neutral 5.0. Now built per-call.
Judge failures were silent. Added health() / assert_healthy() so a degenerate judge halts training instead of quietly flattening the reward.
Prior run's completeness check used straight quotes only; curly ” endings scored 0.3 instead of 1.0. Fixed in gates.ends_cleanly.
Calibration cells A and B originally differed in model and truncation, confounding the measurement. Replaced with a matched cell (cell A's own stories, cut mid-word).
TRL 1.10 removed max_prompt_length (GRPOConfig and DPOConfig) and warmup_ratio (DPOConfig). Caught by a CPU dry-run that constructs every config object before the pipeline needs them, rather than by a stage failing hours in.
m_i <= log(1+eps) ~ 0 is always negative, so gating an ineligible sample to 0.0 in the marginal channel would have handed it the highest diversity credit in its group — a reward-hacking channel of our own construction. Fixed by flooring ineligible samples to the minimum among eligible ones, a rule that is correct regardless of a channel's sign convention. Covered by test_gated_sample_never_outranks_in_marginal_channel.
self-BLEU could exceed 1.0 by ~1e-10 on exact matches due to smoothing; clamped.

Test suite

56 tests across four files, all passing:

file	n	covers
`test_diversity.py`	15	deviation, log-det, marginal contributions, greedy subset, and the E1-vs-E2 hypothesis asserted directly
`test_rewards.py`	10	channel construction, quality gating, tau conditioning, the marginal sign trap, degenerate groups, memoization
`test_pairs.py`	14	DivPO emb/prob selection (incl. the probability inversion), skip logic, E3 multi-positive rows, negative rotation, weight normalization
`test_eval_metrics.py`	17	distinct-4, self-BLEU directionality, cluster count, effective rank, top-k entropy renormalization

Open recommendation: beta (KL coefficient) should be 0 in a future iteration

Measured mid-run on E0 at beta=0.02:

KL penalty term (beta x KL) = 0.000509
|loss| mean                 = 0.118202
-> KL contributes 0.4% of total loss magnitude

Empirically beta=0.02 is already near-inert here. Dropping it to 0.01 or 0.005 would change nothing detectable; the interesting choice is 0 vs non-zero.

The principled argument for 0 is stronger than the magnitude argument. The reference policy is the mode-collapsed distribution under study — base Qwen3-4B has effective rank 2.006 out of 16. A KL penalty regularizes the policy toward that reference, i.e. toward the exact pathology the experiment is trying to escape. In reasoning-RL, KL earns its place by preventing degenerate output; here that job is done better by the programmatic gates (completeness, length window, 4-gram loop detection, entropy floor, non-ASCII), which have no such conflict of interest. Dr. GRPO and much recent work already drop KL entirely.

Not changed for this study, because beta must be identical across arms for E0 to function as a control, and switching now would force a third restart of E0 (at step 123/300) to buy a difference worth 0.4% of the loss. Recorded as the first change to make in a follow-up run.

Related: an earlier note in this log attributed the KL curve's rise-then-settle shape to "the KL penalty pulling the policy back". That was wrong — a term worth 0.4% of the loss cannot do that. The likelier explanation is the policy converging on a reward optimum near the reference.