PEFT How to use CAYTU/whosper-large with PEFT:
from peft import PeftModel
from transformers import AutoModelForSeq2SeqLM
base_model = AutoModelForSeq2SeqLM.from_pretrained("openai/whisper-large-v2")
model = PeftModel.from_pretrained(base_model, "CAYTU/whosper-large")