PEFT How to use acwkim/mod-dpo-safer with PEFT:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("PKU-Alignment/alpaca-7b-reproduced")
model = PeftModel.from_pretrained(base_model, "acwkim/mod-dpo-safer") 
