# Setup actually used, and every deviation from the plan Written during setup, updated as things changed. The point of this file is that the final report should never be the first place a deviation is disclosed. ## Hardware (differs from the brief's assumption) | | assumed | actual | |---|---|---| | GPU | ~48–56 GB (RTX 5090 / 6000 Ada class) | **RTX 5090, 32.6 GB** | | CPU / RAM | — | 32 cores / 188 GB | | Disk | — | 70 GB on `/workspace` | | Driver / CUDA | — | 580.65.06 / CUDA 13.0, sm_120 (Blackwell) | 32 GB rather than ~50 GB is the single most consequential fact in this setup. It is what forced the model split below and every batch-size decision. ## Software | package | version | note | |---|---|---| | torch | 2.13.0+cu130 | upgraded by vLLM; `sm_120` present in arch list | | vLLM | 0.27.1 | **above** TRL's supported range (0.17–0.26); works, warns | | transformers | 5.15.0 | | | trl | 1.10.0 | provides `multi_objective_aggregation` | | peft | 0.20.0 | | | liger-kernel | installed | fused RMSNorm/SwiGLU/RoPE + LM-head CE | | sentence-transformers | 5.7.0 | `BAAI/bge-base-en-v1.5` | ## Deviations from the plan **1. No Unsloth. TRL `GRPOTrainer` + vLLM colocate instead.** The plan specifies Unsloth GRPO. Unsloth pins torch/vLLM versions that conflict with the working sm_120 stack (torch 2.13 + vLLM 0.27.1), and downgrading risks losing Blackwell support entirely — which took real effort to obtain. TRL 1.10's GRPOTrainer supports vLLM colocate natively and additionally supports the exact GDPO aggregation this study needs. Cost: no Unsloth memory savings, which is partly why the batch geometry below is so tight. **2. FlashInfer disabled.** vLLM's bundled FlashInfer misreads the CUDA runtime (`SM 12.x requires CUDA >= 12.9`) and then rejects an sm_120 card as *below* sm_75, killing EngineCore at warm-up. Worked around with `VLLM_ATTENTION_BACKEND=FLASH_ATTN` and `VLLM_USE_FLASHINFER_SAMPLER=0`. Verified equivalent output vs `TRITON_ATTN`. **3. Model split: Qwen3-4B for the frontier, Qwen3-8B for scaling only.** The user asked about "Qwen3.5-9B". Its config shows 24 `linear_attention` layers + 8 `full_attention` (3:1 hybrid, with `mamba_ssm_dtype`), a vision tower, an MTP head and a 248k vocab. It is neither transformer-only nor text-only, and the LoRA-GRPO path through TRL/vLLM for linear-attention layers is unproven. Rejected. `Qwen/Qwen3-8B` is pure `Qwen3ForCausalLM` and is the transformer-only ~9B-class option (there is no `Qwen3-8B-Instruct-2507`). 8B is infeasible for the *online* arms on 32 GB, and this is arithmetic, not preference. GRPO with colocated vLLM needs two resident copies of the weights: ``` Qwen3-8B GRPO: 16.4 (trainer) + 16.4 (vLLM) = 32.8 GB > 32 GB, before any KV cache Qwen3-4B GRPO: 8.05 (trainer) + 8.05 (vLLM) = 16.1 GB ``` LoRA does not help here: it shrinks optimizer state (~98 GB → ~1.2 GB for 8B) but the frozen base still has to be resident. DPO never generates during training, so it needs one copy and gets its reference model free by disabling adapters — which is why 8B DPO fits at ~21 GB while 8B GRPO cannot. Decision: all six primary arms on Qwen3-4B-Instruct-2507 sharing one pool, plus E3-8B/E4-8B on a separate 8B pool as a scaling arm. **4. Judge: per-story absolute scoring, not batched relative scoring.** The prior run sent 16 samples per call and asked for relative scores. Quality is compared against a fixed gate `tau` and across models at eval, so it must be absolutely calibrated. Per-story also makes the cache effective (identical stories recur across arms; each is paid for once). **5. Judge model `deepseek/deepseek-v4-flash-0731` with reasoning DISABLED.** It is a reasoning model; left alone it spends its entire output budget on `reasoning` and returns `content: null`. Calibration caught this at **4/166 calls succeeding**, with every failure silently becoming a neutral 5.0 — a judge that scores everything identically while appearing to work. `reasoning:{enabled:false}` fixes it and is 25× cheaper and 5× faster (30 output tokens vs 764). **6. Judge rubric revised once (v3 → v4).** Allowed by the brief. Strengthened the finishedness check into an explicit first step. See `01_judge_calibration.md`. **7. Judge truncation-sensitivity reclassified as informational.** A_complete − B_cut_matched = +1.33, below the +2.0 I set. Truncation is caught deterministically with 100% recall by `gates.check()`, and only gate-passing stories are ever sent to the judge, so judge truncation sensitivity cannot influence any gradient. The blocking checks (incoherence +2.25, repetition +4.67, non-saturation sd 0.76) all pass. **8. Batch geometry, forced by 32 GB.** | knob | plan | actual | why | |---|---|---|---| | G | 8 | 8 | unchanged | | max_completion_length | 768 | **1024** | *raised*; see below | | per_device_train_batch_size | — | 2 | forward micro-batch; peak is the `2 × seq × 151936` logits tensor | | gradient_accumulation_steps | — | 8 | keeps generation_batch = 16 = 2 prompts × G | | vllm_gpu_memory_utilization | — | 0.33 | ~10.8 GB: 8.05 weights + KV + workspace | Only `generation_batch_size` must be divisible by `num_generations`; the micro-batch need not be. So the forward can be shrunk without changing GRPO group semantics — advantages are still computed over full groups of 8. **9. Token budget RAISED, not lowered.** The plan says 768. A 500-word story is ~650 Qwen tokens, so 768 leaves no headroom and would have re-created the exact truncation bug that killed the prior run. Set to 1024 (~790 words). The length gate rejects anything over 600 words (~780 tokens), so 1024 sits *above the entire accept region*: no story that could ever be scored is truncated by it. Nothing downstream truncates either — the judge and the embedder both receive complete story text. **10. Aggregation is GDPO, and it is native.** The brief allowed a single conditioned scalar as a v1 fallback. Not needed: TRL 1.10's `multi_objective_aggregation="normalize_then_sum"` is a line-for-line implementation of GDPO (arXiv 2601.05242, Liu et al., NVIDIA) — group-wise normalization per reward channel, then batch-wise advantage normalization. E1 and E2 therefore run true decoupled normalization. Consequence: α and γ weight *standardized* channels, so α=0.5 means half an SD of diversity credit per SD of quality, and no manual rescaling of `d_i` into the 0–10 judge range is needed. **11. `n_clusters` as specified does not work; `eff_rank` added as the primary mode metric.** The plan asks for "k-means over embeddings per prompt, silhouette-selected k (crude mode count)". Measured on synthetic 16-point sets, silhouette cannot separate collapse from spread at all: | configuration | best k | silhouette | |---|---|---| | fully collapsed (noise 0.001) | 4 | 0.202 | | fully spread (random) | 7 | 0.195 | | two clear modes | 2 | **0.976** | k-means partitions isotropic data regardless of spread, so the metric would have reported ~4 modes for collapsed output and ~7 for diverse output — fabricated structure, in a headline column. Two changes: `SILHOUETTE_MIN` raised 0.05 → 0.50, making `n_clusters` a *conservative* count of well-separated modes (returns 1 unless structure is unmistakable); and **effective rank** (exp-entropy of the Gram spectrum, Roy & Vetterli) added as the primary continuous mode measure — 1.00 identical, 2.12 two clusters, 13.29 spread, with no threshold to tune. **12. Liger disabled to recover the entropy metric.** `use_liger_kernel=True` routes TRL to `compute_liger_loss`, which logs only `[clip_ratio, kl]` — the `entropy` metric exists solely on the `_compute_loss` path. Liger had been enabled to fix an OOM at micro-batch 2; micro-batch is now 1 (halving the logits tensor), so it is no longer needed. vLLM's budget trimmed 0.32 → 0.29 to cover the difference. Per-token policy entropy cannot be recovered from a finished run, and entropy collapse is a first-class failure signal, so this trade was worth the memory. **13. Entropy is treated asymmetrically.** A large *fall* is strong evidence the policy is going deterministic and is flagged as tainting any diversity claim in that run. A *rise* is only permissive — a policy can raise per-token entropy while staying inside one narrative mode. The prior run demonstrates the dissociation directly (surface variation up, semantic diversity down). Entropy is never optimized. **14. Learning rate raised 2e-6 → 3e-6.** Smoke showed KL ≈ 6e-4 at step 6 under 2e-6; across 300 steps that risked a null result in *every* arm, which would be uninformative. 3e-6 is still inside the plan's 1e-6..5e-6 band. Too-high instead risks instability, which the guardrail catches and which is itself a finding. **15. Eval runs one subprocess per model.** vLLM v1 keeps EngineCore in a child process and `del llm` does not reliably reclaim its GPU memory, so a 7-model in-process loop would OOM on model 2 after model 1 succeeded. Process isolation makes teardown unconditional and stops one model's crash from taking down the whole eval; completed models are cached as row files and skipped on re-run. ## Fallbacks taken (per the plan's ordered list) - G 8→6: **not taken** - max_new_tokens 768→512: **not taken** (raised to 1024 instead) - steps 300→200: not taken yet - policy 4B: taken as the primary, by the VRAM argument above - eval prompts 50→30: not taken ## Bugs found and fixed during setup 1. `asyncio.Semaphore` built in `__init__` binds to the first event loop; `score_many_sync()` calls `asyncio.run()` once per training step, so it would have raised "bound to a different event loop" from step 2 onward and turned every judge call into a neutral 5.0. Now built per-call. 2. Judge failures were silent. Added `health()` / `assert_healthy()` so a degenerate judge halts training instead of quietly flattening the reward. 3. Prior run's completeness check used straight quotes only; curly `”` endings scored 0.3 instead of 1.0. Fixed in `gates.ends_cleanly`. 4. Calibration cells A and B originally differed in model *and* truncation, confounding the measurement. Replaced with a matched cell (cell A's own stories, cut mid-word). 5. TRL 1.10 removed `max_prompt_length` (GRPOConfig *and* DPOConfig) and `warmup_ratio` (DPOConfig). Caught by a CPU dry-run that constructs every config object before the pipeline needs them, rather than by a stage failing hours in. 6. `m_i <= log(1+eps) ~ 0` is always negative, so gating an ineligible sample to `0.0` in the marginal channel would have handed it the *highest* diversity credit in its group — a reward-hacking channel of our own construction. Fixed by flooring ineligible samples to the minimum among eligible ones, a rule that is correct regardless of a channel's sign convention. Covered by `test_gated_sample_never_outranks_in_marginal_channel`. 7. self-BLEU could exceed 1.0 by ~1e-10 on exact matches due to smoothing; clamped. ## Test suite 56 tests across four files, all passing: | file | n | covers | |---|---|---| | `test_diversity.py` | 15 | deviation, log-det, marginal contributions, greedy subset, and the E1-vs-E2 hypothesis asserted directly | | `test_rewards.py` | 10 | channel construction, quality gating, tau conditioning, the marginal sign trap, degenerate groups, memoization | | `test_pairs.py` | 14 | DivPO emb/prob selection (incl. the probability inversion), skip logic, E3 multi-positive rows, negative rotation, weight normalization | | `test_eval_metrics.py` | 17 | distinct-4, self-BLEU directionality, cluster count, effective rank, top-k entropy renormalization | ## Open recommendation: beta (KL coefficient) should be 0 in a future iteration Measured mid-run on E0 at beta=0.02: ``` KL penalty term (beta x KL) = 0.000509 |loss| mean = 0.118202 -> KL contributes 0.4% of total loss magnitude ``` **Empirically beta=0.02 is already near-inert here.** Dropping it to 0.01 or 0.005 would change nothing detectable; the interesting choice is 0 vs non-zero. **The principled argument for 0 is stronger than the magnitude argument.** The reference policy *is* the mode-collapsed distribution under study — base Qwen3-4B has effective rank 2.006 out of 16. A KL penalty regularizes the policy *toward* that reference, i.e. toward the exact pathology the experiment is trying to escape. In reasoning-RL, KL earns its place by preventing degenerate output; here that job is done better by the programmatic gates (completeness, length window, 4-gram loop detection, entropy floor, non-ASCII), which have no such conflict of interest. Dr. GRPO and much recent work already drop KL entirely. **Not changed for this study**, because beta must be identical across arms for E0 to function as a control, and switching now would force a third restart of E0 (at step 123/300) to buy a difference worth 0.4% of the loss. Recorded as the first change to make in a follow-up run. Related: an earlier note in this log attributed the KL curve's rise-then-settle shape to "the KL penalty pulling the policy back". That was wrong — a term worth 0.4% of the loss cannot do that. The likelier explanation is the policy converging on a reward optimum near the reference.