research/12-altered-model-rl-critique.md

# Critique: SDPO + Dr.GRPO + trace-replay-DPO on a personality-altered model

## Bottom line
The proposed three-channel stage is scientifically interesting but unsafe as a first *interpretable* run. Applied to an LMA personality-altered SFT model, the combined recipe confounds at least four effects: task RL, self-distillation of altered reasoning, frontier-teacher imitation, and KL anchoring. If capability or moral-scenarios behavior changes, the result will not identify whether RL washed out, preserved, or amplified the alteration.

## 1. SDPO against the altered model's own hint-conditioned passes
SDPO with the altered model as its own hint-conditioned teacher is only sound if the hints expose latent correct reasoning that the base forward pass underuses. On an altered model, that assumption is fragile. The hint-conditioned pass may instead expose the same distorted policy under a more verbose or more confident trajectory. Then SDPO becomes a consistency regularizer around an already-biased teacher, not an improvement signal.

Main risks:

- **Degenerate fixed point:** teacher and student are same-family, same checkpoint, and differ only by prompting. If hints do not add independent information, the optimum is to imitate the altered model's own conditional distribution.
- **Amplification:** if the altered model has class-specific moral-scenarios distortion, hinting may increase rationalization of the distorted answer. SDPO can convert a soft bias into a sharper preference.
- **Mode collapse / reduced diversity:** KL-to-own-hinted-output rewards low-entropy agreement. Combined with answer-only GRPO, this can produce brittle letter-pattern policies.
- **False preservation:** staying close to the altered hint teacher may look like preserving the personality alteration while actually preserving only task-format artifacts.

So SDPO-only is not a benign auxiliary loss here; it is the channel most likely to test the amplification hypothesis. Treat it as an experimental intervention, not a default stabilizer.

## 2. Hyperparameters and attribution
Do **not** start with alpha_sdpo=0.2 and beta_replay=0.4 in a combined run. Those weights are large enough that any observed outcome will be uninterpretable. The first real run should isolate channels:

1. **GRPO-only baseline:** alpha_sdpo=0, beta_replay=0.
2. **SDPO-only add-on:** alpha_sdpo small, beta_replay=0.
3. **Replay-DPO-only add-on:** alpha_sdpo=0, beta_replay small.
4. **Combined only after the above:** use the smallest weights that showed useful signal without personality drift or distortion amplification.

Safe initial defaults, assuming loss terms are normalized to comparable token/batch scales:

- `alpha_sdpo`: **0.02** for first SDPO run; sweep `{0.0, 0.02, 0.05}`. Avoid 0.2 until there is evidence no amplification occurs.
- `beta_replay`: **0.05** for first replay run; sweep `{0.0, 0.05, 0.10}`. Avoid 0.4 initially because frontier teachers may wash out the alteration and dominate GRPO.
- Combined pilot: `alpha_sdpo=0.02`, `beta_replay=0.05`, only after isolated runs.
- KL coefficient to frozen altered SFT init: `kl_beta=0.02` initial, adaptive to target roughly **0.01-0.03 nats/token** on answer+reasoning tokens. Hard-stop or reduce LR if KL exceeds ~0.08 nats/token or if personality probes drift sharply.
- Also log KL to the original unaltered base/SFT. Do not optimize this KL unless the goal is explicitly de-alteration; use it to measure washout.

## 3. Reward design to avoid MMLU letter hacking
A letter-correctness reward is too hackable unless the interface is constrained.

Recommended reward:

- Require structured output: `{"answer":"A|B|C|D","rationale":"..."}` or a single final `Answer: X` after reasoning.
- Parse only the final answer field; reject multiple final answers.
- Reward: `+1` correct, `0` incorrect, `-0.2` invalid/unparseable, `-0.1` multiple answers, small length penalty after a rationale cap.
- Randomize option order per epoch and track original label mapping.
- Balance batches across subject and moral-scenarios classes, especially the collapsed class-3 subset.
- Hold out a moral-scenarios diagnostic split that is never used for reward.
- Add calibration metrics: entropy over options, answer distribution, invalid rate, rationale length, and per-class accuracy. A model that learns “always C” or exploits option priors should be obvious.

Do not reward chain-of-thought style itself. If rationales are used, score only final answer and maybe format validity; otherwise the model can learn persuasive distorted rationalizations.

## 4. Cheapest experiment that distinguishes washout / preserve / amplify
Use one altered checkpoint, one matched unaltered checkpoint, and a fixed evaluation harness. Run short, equal-token pilots with identical prompts/seeds:

- A0: altered SFT, no RL.
- A1: GRPO-only.
- A2: GRPO + SDPO small.
- A3: GRPO + replay-DPO small.
- A4: combined small, only if A1-A3 are interpretable.

Evaluate before/after on: general MMLU, moral_scenarios by class, LMA personality/psychiatric probes, KL to altered init, KL to unaltered base, option distribution, and refusal/format invalid rates.

Interpretation:

- **Washout:** capability improves while personality markers and altered-specific moral signature move toward unaltered baseline.
- **Preservation:** capability improves with stable personality markers and stable moral-scenarios signature.
- **Amplification:** moral class-3 or cognitive-distortion probes worsen, confidence/entropy sharpens, or SDPO run diverges more than GRPO-only.

The proposed alpha=0.2/beta=0.4 combined recipe should be considered a later stress test, not a first run.