research/12-altered-model-rl-critique.md

Critique: SDPO + Dr.GRPO + trace-replay-DPO on a personality-altered model

Bottom line

The proposed three-channel stage is scientifically interesting but unsafe as a first interpretable run. Applied to an LMA personality-altered SFT model, the combined recipe confounds at least four effects: task RL, self-distillation of altered reasoning, frontier-teacher imitation, and KL anchoring. If capability or moral-scenarios behavior changes, the result will not identify whether RL washed out, preserved, or amplified the alteration.

1. SDPO against the altered model's own hint-conditioned passes

SDPO with the altered model as its own hint-conditioned teacher is only sound if the hints expose latent correct reasoning that the base forward pass underuses. On an altered model, that assumption is fragile. The hint-conditioned pass may instead expose the same distorted policy under a more verbose or more confident trajectory. Then SDPO becomes a consistency regularizer around an already-biased teacher, not an improvement signal.

Main risks:

• Degenerate fixed point: teacher and student are same-family, same checkpoint, and differ only by prompting. If hints do not add independent information, the optimum is to imitate the altered model's own conditional distribution.
• Amplification: if the altered model has class-specific moral-scenarios distortion, hinting may increase rationalization of the distorted answer. SDPO can convert a soft bias into a sharper preference.
• Mode collapse / reduced diversity: KL-to-own-hinted-output rewards low-entropy agreement. Combined with answer-only GRPO, this can produce brittle letter-pattern policies.
• False preservation: staying close to the altered hint teacher may look like preserving the personality alteration while actually preserving only task-format artifacts.

So SDPO-only is not a benign auxiliary loss here; it is the channel most likely to test the amplification hypothesis. Treat it as an experimental intervention, not a default stabilizer.

2. Hyperparameters and attribution

Do not start with alpha_sdpo=0.2 and beta_replay=0.4 in a combined run. Those weights are large enough that any observed outcome will be uninterpretable. The first real run should isolate channels:

1. GRPO-only baseline: alpha_sdpo=0, beta_replay=0.
2. SDPO-only add-on: alpha_sdpo small, beta_replay=0.
3. Replay-DPO-only add-on: alpha_sdpo=0, beta_replay small.
4. Combined only after the above: use the smallest weights that showed useful signal without personality drift or distortion amplification.

Safe initial defaults, assuming loss terms are normalized to comparable token/batch scales:

• alpha_sdpo: 0.02 for first SDPO run; sweep {0.0, 0.02, 0.05}. Avoid 0.2 until there is evidence no amplification occurs.
• beta_replay: 0.05 for first replay run; sweep {0.0, 0.05, 0.10}. Avoid 0.4 initially because frontier teachers may wash out the alteration and dominate GRPO.
• Combined pilot: alpha_sdpo=0.02, beta_replay=0.05, only after isolated runs.
• KL coefficient to frozen altered SFT init: kl_beta=0.02 initial, adaptive to target roughly 0.01-0.03 nats/token on answer+reasoning tokens. Hard-stop or reduce LR if KL exceeds ~0.08 nats/token or if personality probes drift sharply.
• Also log KL to the original unaltered base/SFT. Do not optimize this KL unless the goal is explicitly de-alteration; use it to measure washout.

3. Reward design to avoid MMLU letter hacking

A letter-correctness reward is too hackable unless the interface is constrained.

Recommended reward:

• Require structured output: {"answer":"A|B|C|D","rationale":"..."} or a single final Answer: X after reasoning.
• Parse only the final answer field; reject multiple final answers.
• Reward: +1 correct, 0 incorrect, -0.2 invalid/unparseable, -0.1 multiple answers, small length penalty after a rationale cap.
• Randomize option order per epoch and track original label mapping.
• Balance batches across subject and moral-scenarios classes, especially the collapsed class-3 subset.
• Hold out a moral-scenarios diagnostic split that is never used for reward.
• Add calibration metrics: entropy over options, answer distribution, invalid rate, rationale length, and per-class accuracy. A model that learns “always C” or exploits option priors should be obvious.

Do not reward chain-of-thought style itself. If rationales are used, score only final answer and maybe format validity; otherwise the model can learn persuasive distorted rationalizations.

4. Cheapest experiment that distinguishes washout / preserve / amplify

Use one altered checkpoint, one matched unaltered checkpoint, and a fixed evaluation harness. Run short, equal-token pilots with identical prompts/seeds:

• A0: altered SFT, no RL.
• A1: GRPO-only.
• A2: GRPO + SDPO small.
• A3: GRPO + replay-DPO small.
• A4: combined small, only if A1-A3 are interpretable.

Evaluate before/after on: general MMLU, moral_scenarios by class, LMA personality/psychiatric probes, KL to altered init, KL to unaltered base, option distribution, and refusal/format invalid rates.

Interpretation:

• Washout: capability improves while personality markers and altered-specific moral signature move toward unaltered baseline.
• Preservation: capability improves with stable personality markers and stable moral-scenarios signature.
• Amplification: moral class-3 or cognitive-distortion probes worsen, confidence/entropy sharpens, or SDPO run diverges more than GRPO-only.

The proposed alpha=0.2/beta=0.4 combined recipe should be considered a later stress test, not a first run.