SetonLiang2/qwen25-7b-assignment4-dpo-adapter

Qwen2.5-7B DPO Adapter (Assignment 4)

This is a PEFT LoRA adapter fine-tuned from Qwen/Qwen2.5-7B-Instruct for a course assignment on DPO-based preference optimization.

Quick Start

Option 1: Transformers + PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_path = "SetonLiang2/qwen25-7b-assignment4-dpo-adapter"

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()

messages = [{"role": "user", "content": "请解释什么是DPO"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

Option 2: Merge Before Inference (Optional)

If your deployment environment does not support PEFT, you can merge the adapter into the base model first and run inference on the merged model.

Training Summary

• Base model: Qwen/Qwen2.5-7B-Instruct
• Method: DPO (TRL) + LoRA (PEFT)
• Preference data: chosen/rejected pairs constructed from multi-candidate responses using PairRM

Intended Use

• Research and coursework experiments
• Comparing base model outputs vs DPO-tuned outputs

Limitations

• Small-scale training data (coursework setting), so generalization is limited
• The model may still produce unstable or incomplete responses

Framework versions

• PEFT 0.15.1