logs/experiments/02_setup_and_deviations.md

# Setup actually used, and every deviation from the plan

Written during setup, updated as things changed. The point of this file is that
the final report should never be the first place a deviation is disclosed.

## Hardware (differs from the brief's assumption)

| | assumed | actual |
|---|---|---|
| GPU | ~48–56 GB (RTX 5090 / 6000 Ada class) | **RTX 5090, 32.6 GB** |
| CPU / RAM | — | 32 cores / 188 GB |
| Disk | — | 70 GB on `/workspace` |
| Driver / CUDA | — | 580.65.06 / CUDA 13.0, sm_120 (Blackwell) |

32 GB rather than ~50 GB is the single most consequential fact in this setup.
It is what forced the model split below and every batch-size decision.

## Software

| package | version | note |
|---|---|---|
| torch | 2.13.0+cu130 | upgraded by vLLM; `sm_120` present in arch list |
| vLLM | 0.27.1 | **above** TRL's supported range (0.17–0.26); works, warns |
| transformers | 5.15.0 | |
| trl | 1.10.0 | provides `multi_objective_aggregation` |
| peft | 0.20.0 | |
| liger-kernel | installed | fused RMSNorm/SwiGLU/RoPE + LM-head CE |
| sentence-transformers | 5.7.0 | `BAAI/bge-base-en-v1.5` |

## Deviations from the plan

**1. No Unsloth. TRL `GRPOTrainer` + vLLM colocate instead.**
The plan specifies Unsloth GRPO. Unsloth pins torch/vLLM versions that conflict
with the working sm_120 stack (torch 2.13 + vLLM 0.27.1), and downgrading risks
losing Blackwell support entirely — which took real effort to obtain. TRL 1.10's
GRPOTrainer supports vLLM colocate natively and additionally supports the exact
GDPO aggregation this study needs. Cost: no Unsloth memory savings, which is
partly why the batch geometry below is so tight.

**2. FlashInfer disabled.**
vLLM's bundled FlashInfer misreads the CUDA runtime (`SM 12.x requires
CUDA >= 12.9`) and then rejects an sm_120 card as *below* sm_75, killing
EngineCore at warm-up. Worked around with `VLLM_ATTENTION_BACKEND=FLASH_ATTN`
and `VLLM_USE_FLASHINFER_SAMPLER=0`. Verified equivalent output vs `TRITON_ATTN`.

**3. Model split: Qwen3-4B for the frontier, Qwen3-8B for scaling only.**
The user asked about "Qwen3.5-9B". Its config shows 24 `linear_attention` layers
+ 8 `full_attention` (3:1 hybrid, with `mamba_ssm_dtype`), a vision tower, an MTP
head and a 248k vocab. It is neither transformer-only nor text-only, and the
LoRA-GRPO path through TRL/vLLM for linear-attention layers is unproven. Rejected.
`Qwen/Qwen3-8B` is pure `Qwen3ForCausalLM` and is the transformer-only ~9B-class
option (there is no `Qwen3-8B-Instruct-2507`).

8B is infeasible for the *online* arms on 32 GB, and this is arithmetic, not
preference. GRPO with colocated vLLM needs two resident copies of the weights:

```
Qwen3-8B GRPO:  16.4 (trainer) + 16.4 (vLLM) = 32.8 GB > 32 GB, before any KV cache
Qwen3-4B GRPO:   8.05 (trainer) + 8.05 (vLLM) = 16.1 GB
```

LoRA does not help here: it shrinks optimizer state (~98 GB → ~1.2 GB for 8B)
but the frozen base still has to be resident. DPO never generates during
training, so it needs one copy and gets its reference model free by disabling
adapters — which is why 8B DPO fits at ~21 GB while 8B GRPO cannot.

Decision: all six primary arms on Qwen3-4B-Instruct-2507 sharing one pool, plus
E3-8B/E4-8B on a separate 8B pool as a scaling arm.

**4. Judge: per-story absolute scoring, not batched relative scoring.**
The prior run sent 16 samples per call and asked for relative scores. Quality is
compared against a fixed gate `tau` and across models at eval, so it must be
absolutely calibrated. Per-story also makes the cache effective (identical
stories recur across arms; each is paid for once).

**5. Judge model `deepseek/deepseek-v4-flash-0731` with reasoning DISABLED.**
It is a reasoning model; left alone it spends its entire output budget on
`reasoning` and returns `content: null`. Calibration caught this at **4/166 calls
succeeding**, with every failure silently becoming a neutral 5.0 — a judge that
scores everything identically while appearing to work. `reasoning:{enabled:false}`
fixes it and is 25× cheaper and 5× faster (30 output tokens vs 764).

**6. Judge rubric revised once (v3 → v4).** Allowed by the brief. Strengthened
the finishedness check into an explicit first step. See
`01_judge_calibration.md`.

**7. Judge truncation-sensitivity reclassified as informational.**
A_complete − B_cut_matched = +1.33, below the +2.0 I set. Truncation is caught
deterministically with 100% recall by `gates.check()`, and only gate-passing
stories are ever sent to the judge, so judge truncation sensitivity cannot
influence any gradient. The blocking checks (incoherence +2.25, repetition
+4.67, non-saturation sd 0.76) all pass.

**8. Batch geometry, forced by 32 GB.**

| knob | plan | actual | why |
|---|---|---|---|
| G | 8 | 8 | unchanged |
| max_completion_length | 768 | **1024** | *raised*; see below |
| per_device_train_batch_size | — | 2 | forward micro-batch; peak is the `2 × seq × 151936` logits tensor |
| gradient_accumulation_steps | — | 8 | keeps generation_batch = 16 = 2 prompts × G |
| vllm_gpu_memory_utilization | — | 0.33 | ~10.8 GB: 8.05 weights + KV + workspace |

Only `generation_batch_size` must be divisible by `num_generations`; the
micro-batch need not be. So the forward can be shrunk without changing GRPO
group semantics — advantages are still computed over full groups of 8.

**9. Token budget RAISED, not lowered.** The plan says 768. A 500-word story is
~650 Qwen tokens, so 768 leaves no headroom and would have re-created the exact
truncation bug that killed the prior run. Set to 1024 (~790 words). The length
gate rejects anything over 600 words (~780 tokens), so 1024 sits *above the
entire accept region*: no story that could ever be scored is truncated by it.
Nothing downstream truncates either — the judge and the embedder both receive
complete story text.

**10. Aggregation is GDPO, and it is native.** The brief allowed a single
conditioned scalar as a v1 fallback. Not needed: TRL 1.10's
`multi_objective_aggregation="normalize_then_sum"` is a line-for-line
implementation of GDPO (arXiv 2601.05242, Liu et al., NVIDIA) — group-wise
normalization per reward channel, then batch-wise advantage normalization. E1
and E2 therefore run true decoupled normalization. Consequence: α and γ weight
*standardized* channels, so α=0.5 means half an SD of diversity credit per SD of
quality, and no manual rescaling of `d_i` into the 0–10 judge range is needed.

**11. `n_clusters` as specified does not work; `eff_rank` added as the primary
mode metric.** The plan asks for "k-means over embeddings per prompt,
silhouette-selected k (crude mode count)". Measured on synthetic 16-point sets,
silhouette cannot separate collapse from spread at all:

| configuration | best k | silhouette |
|---|---|---|
| fully collapsed (noise 0.001) | 4 | 0.202 |
| fully spread (random) | 7 | 0.195 |
| two clear modes | 2 | **0.976** |

k-means partitions isotropic data regardless of spread, so the metric would have
reported ~4 modes for collapsed output and ~7 for diverse output — fabricated
structure, in a headline column. Two changes: `SILHOUETTE_MIN` raised 0.05 →
0.50, making `n_clusters` a *conservative* count of well-separated modes
(returns 1 unless structure is unmistakable); and **effective rank**
(exp-entropy of the Gram spectrum, Roy & Vetterli) added as the primary
continuous mode measure — 1.00 identical, 2.12 two clusters, 13.29 spread, with
no threshold to tune.

**12. Liger disabled to recover the entropy metric.** `use_liger_kernel=True`
routes TRL to `compute_liger_loss`, which logs only `[clip_ratio, kl]` — the
`entropy` metric exists solely on the `_compute_loss` path. Liger had been
enabled to fix an OOM at micro-batch 2; micro-batch is now 1 (halving the logits
tensor), so it is no longer needed. vLLM's budget trimmed 0.32 → 0.29 to cover
the difference. Per-token policy entropy cannot be recovered from a finished
run, and entropy collapse is a first-class failure signal, so this trade was
worth the memory.

**13. Entropy is treated asymmetrically.** A large *fall* is strong evidence the
policy is going deterministic and is flagged as tainting any diversity claim in
that run. A *rise* is only permissive — a policy can raise per-token entropy
while staying inside one narrative mode. The prior run demonstrates the
dissociation directly (surface variation up, semantic diversity down). Entropy
is never optimized.

**14. Learning rate raised 2e-6 → 3e-6.** Smoke showed KL ≈ 6e-4 at step 6 under
2e-6; across 300 steps that risked a null result in *every* arm, which would be
uninformative. 3e-6 is still inside the plan's 1e-6..5e-6 band. Too-high instead
risks instability, which the guardrail catches and which is itself a finding.

**15. Eval runs one subprocess per model.** vLLM v1 keeps EngineCore in a child
process and `del llm` does not reliably reclaim its GPU memory, so a 7-model
in-process loop would OOM on model 2 after model 1 succeeded. Process isolation
makes teardown unconditional and stops one model's crash from taking down the
whole eval; completed models are cached as row files and skipped on re-run.

## Fallbacks taken (per the plan's ordered list)

- G 8→6: **not taken**
- max_new_tokens 768→512: **not taken** (raised to 1024 instead)
- steps 300→200: not taken yet
- policy 4B: taken as the primary, by the VRAM argument above
- eval prompts 50→30: not taken

## Bugs found and fixed during setup

1. `asyncio.Semaphore` built in `__init__` binds to the first event loop;
   `score_many_sync()` calls `asyncio.run()` once per training step, so it would
   have raised "bound to a different event loop" from step 2 onward and turned
   every judge call into a neutral 5.0. Now built per-call.
2. Judge failures were silent. Added `health()` / `assert_healthy()` so a
   degenerate judge halts training instead of quietly flattening the reward.
3. Prior run's completeness check used straight quotes only; curly `”` endings
   scored 0.3 instead of 1.0. Fixed in `gates.ends_cleanly`.
4. Calibration cells A and B originally differed in model *and* truncation,
   confounding the measurement. Replaced with a matched cell (cell A's own
   stories, cut mid-word).
5. TRL 1.10 removed `max_prompt_length` (GRPOConfig *and* DPOConfig) and
   `warmup_ratio` (DPOConfig). Caught by a CPU dry-run that constructs every
   config object before the pipeline needs them, rather than by a stage failing
   hours in.
6. `m_i <= log(1+eps) ~ 0` is always negative, so gating an ineligible sample to
   `0.0` in the marginal channel would have handed it the *highest* diversity
   credit in its group — a reward-hacking channel of our own construction. Fixed
   by flooring ineligible samples to the minimum among eligible ones, a rule
   that is correct regardless of a channel's sign convention. Covered by
   `test_gated_sample_never_outranks_in_marginal_channel`.
7. self-BLEU could exceed 1.0 by ~1e-10 on exact matches due to smoothing;
   clamped.

## Test suite

56 tests across four files, all passing:

| file | n | covers |
|---|---|---|
| `test_diversity.py` | 15 | deviation, log-det, marginal contributions, greedy subset, and the E1-vs-E2 hypothesis asserted directly |
| `test_rewards.py` | 10 | channel construction, quality gating, tau conditioning, the marginal sign trap, degenerate groups, memoization |
| `test_pairs.py` | 14 | DivPO emb/prob selection (incl. the probability inversion), skip logic, E3 multi-positive rows, negative rotation, weight normalization |
| `test_eval_metrics.py` | 17 | distinct-4, self-BLEU directionality, cluster count, effective rank, top-k entropy renormalization |

## Open recommendation: beta (KL coefficient) should be 0 in a future iteration

Measured mid-run on E0 at beta=0.02:

```
KL penalty term (beta x KL) = 0.000509
|loss| mean                 = 0.118202
-> KL contributes 0.4% of total loss magnitude
```

**Empirically beta=0.02 is already near-inert here.** Dropping it to 0.01 or
0.005 would change nothing detectable; the interesting choice is 0 vs non-zero.

**The principled argument for 0 is stronger than the magnitude argument.** The
reference policy *is* the mode-collapsed distribution under study — base
Qwen3-4B has effective rank 2.006 out of 16. A KL penalty regularizes the policy
*toward* that reference, i.e. toward the exact pathology the experiment is
trying to escape. In reasoning-RL, KL earns its place by preventing degenerate
output; here that job is done better by the programmatic gates (completeness,
length window, 4-gram loop detection, entropy floor, non-ASCII), which have no
such conflict of interest. Dr. GRPO and much recent work already drop KL
entirely.

**Not changed for this study**, because beta must be identical across arms for
E0 to function as a control, and switching now would force a third restart of
E0 (at step 123/300) to buy a difference worth 0.4% of the loss. Recorded as the
first change to make in a follow-up run.

Related: an earlier note in this log attributed the KL curve's rise-then-settle
shape to "the KL penalty pulling the policy back". That was wrong — a term worth
0.4% of the loss cannot do that. The likelier explanation is the policy
converging on a reward optimum near the reference.