YashNagraj75/RLHF-Flant5

RLHF Flan-T5: Low-Toxicity Dialogue Summarization

A fine-tuned version of google/flan-t5-base trained to produce concise, low-toxicity summaries of conversational dialogues. Training followed a two-stage pipeline: supervised fine-tuning with LoRA, followed by reinforcement learning from human feedback (RLHF) using PPO with a toxicity classifier as the reward signal.

Training Pipeline

Stage 1 — Supervised Fine-Tuning with LoRA

The base model was first fine-tuned on the DialogSum dataset using PEFT LoRA.

Parameter	Value
Base model	google/flan-t5-base
Dataset	knkarthick/dialogsum (train split, filtered 200–1000 chars)
Task type	SEQ_2_SEQ_LM
LoRA rank (r)	32
LoRA alpha	32
Target modules	q, v
Dropout	0.05
Adapter saved to	flan-t5-lora/

Stage 2 — RLHF with PPO

The LoRA-adapted model was wrapped with a value head and trained with PPO using a hate-speech detection model as the reward signal.

Parameter	Value
Reward model	facebook/roberta-hate-speech-dynabench-r4-target
PPO library	trl (AutoModelForSeq2SeqLMWithValueHead)
Reference model	Frozen copy of the SFT model (create_reference_model)
Reward metric	Toxicity score (lower = better)
Adapter saved to	Flant5-RLHF/

How to Use

Load the RLHF adapter (recommended)

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
model = PeftModel.from_pretrained(base_model, "YashNagraj75/RLHF-Flant5/Flant5-RLHF")
tokenizer = AutoTokenizer.from_pretrained("YashNagraj75/RLHF-Flant5")

dialogue = "Person A: ...\nPerson B: ..."
prompt = f"Summarize the following conversation.\n\n{dialogue}\n\nSummary:"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Load the SFT-only adapter

model = PeftModel.from_pretrained(base_model, "YashNagraj75/RLHF-Flant5/flan-t5-lora")

Repository Contents

File	Description
RLHF-flant5.py	Full training script (SFT + PPO RLHF loop)
Flant5-RLHF/	RLHF-tuned LoRA adapter (safetensors + config)
flan-t5-lora/	SFT-only LoRA adapter

Framework Versions

• PyTorch, Transformers, PEFT, TRL, Evaluate

License

MIT