harsh9/persona-collapse-ref-sft-olmo3-7b

Ref-SFT for Olmo-3-7B-Instruct

LoRA adapter for allenai/Olmo-3-7B-Instruct from the paper Diagnosing and Repairing Persona Collapse in LLM Advice (arXiv:2607.08326).

Models tend to answer every advice request with the same reassuring persona. Ref-SFT teaches the model to pick an approach that fits the situation: comfort, challenge, or practical guidance. It learns from model-written replies, not from the human replies themselves.

For each training situation, Olmo-3-7B-Instruct wrote six candidate replies: three ordinary ones and three guided by a rationale reconstructed from the top human reply. Gemma-4-31B-it compared each candidate with the human reply on hedonic tone and agency support. We fine-tune on the candidate closest to the human reply. At deployment the model sees only the situation.

The paper's best recipe starts from this adapter and adds quality-reward RL: harsh9/persona-collapse-ref-sft-quality-grpo-olmo3-7b.

Use

The model was trained and evaluated with this system prompt and the raw post as the user message.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "allenai/Olmo-3-7B-Instruct"
tok = AutoTokenizer.from_pretrained(base_id)
model = AutoModelForCausalLM.from_pretrained(base_id, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "harsh9/persona-collapse-ref-sft-olmo3-7b")

system = "Read the SITUATION and write a short, concise response in a single paragraph. Do not write JSON."
post = "My roommate keeps eating my food. I've hinted a few times but nothing changes. What should I do?"
messages = [{"role": "system", "content": system}, {"role": "user", "content": post}]
ids = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**ids, do_sample=True, temperature=0.7, top_p=0.9, max_new_tokens=384)
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=True))

The paper's evaluation samples at temperature 0.7 and top-p 0.9 with at most 384 new tokens.

Training

• Base: allenai/Olmo-3-7B-Instruct.
• Data: 1,825 training situations. They come from 6,610 training situations (Reddit, Stack Exchange, CounselChat, CareerNet) and are the ones with a clearly closest candidate and a clearly farther one. 96 more are held out for validation.
• Of these 1,921 selected replies, 1,801 were written with rationale guidance and 120 without. The training target is always model-written text.
• LoRA rank 32, alpha 64, dropout 0.05, all linear layers.
• 3 epochs, learning rate 2e-4, cosine decay, 3% warmup, effective batch 16, max length 4,096. Checkpoint chosen by validation loss.
• Training seed 42. The paper also reports seeds 1 and 2.

Results

Model-judge results on 1,652 held-out situations, averaged over Gemma-4-31B-it and Qwen3.5-27B. Numbers are from the paper's combined-judge overview table.

Model	Persona fit	Warranted pushback	Unwarranted pushback	Δ truth	Δ harm
Olmo-3-7B-Instruct	23.6	13.5	2.0	0.00	0.00
IPD-SFT (imitation)	16.0	55.5	11.5	-0.80	+1.18
Ref-SFT (this adapter)	32.6	39.5	4.5	+0.40	-0.20
Quality-GRPO	31.4	28.6	2.8	+0.64	-0.32
Ref-SFT → Quality-GRPO	40.7	51.0	6.4	+0.98	-0.36

• Persona fit (%): the reply's persona matches the human reply's persona. Enabling targets count as supportive and cynical targets as challenging.
• Warranted / unwarranted pushback (%): the reply challenges the person in situations where challenge is or is not called for.
• Δ truth and Δ harm: change from Instruct in 1-7 ratings of truth/accountability and potential harm. Higher truth and lower harm are better.
• Ref-SFT and the GRPO rows are means over three training seeds. This repo holds seed 42.

In the paper's second human study (N=159), Ref-SFT was near parity with Instruct on immediate preference.

Limitations

This is a research checkpoint from a study of advice style. It was evaluated with model judges and a short human rating study, not with people seeking help. Do not use it as a counseling or crisis service. Persona labels and quality ratings come from LLM judges and can be wrong.

Citation

@misc{kumar2026persona,
  title         = {Diagnosing and Repairing Persona Collapse in LLM Advice},
  author        = {Kumar, Harsh and Vold, Karina and Tay, Louis and Anderson, Ashton},
  year          = {2026},
  eprint        = {2607.08326},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CY},
  url           = {https://arxiv.org/abs/2607.08326}
}