Fine tuning with LORA
I am trying to fine tune blip2 with image as input as text as output with the following code
quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16)
processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b", quantization_config=quantization_config, device_map={'':0})
return model, processor
config = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.10, bias="none", target_modules=["q_proj", "k_proj"])
model, processor = get_model_processor(model)
model = get_peft_model(model, config)
model.print_trainable_parameters()
I have been playing around with r and lora_alpha values and got "ok" results. Does anyone have experience setting these values? . Also what would the task_type be for the LoraConfig.
hi, Could you please make all the codes public? I'm currently working on fine-tune blip2 on the vqa task, thank you.
Refer to the demo notebooks here: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2
Refer to the demo notebooks here: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/BLIP-2
hi, I looked at it, but I found that it was for the caption task, and I tried to change it to the vqa task, but it failed.
How should we fine-tune this model for VQA. The above link is helpful for inference.
I found this resource but it is for BLIP : https://github.com/dino-chiio/blip-vqa-finetune/blob/main/finetuning.py
I need to know while fine-tuning do we pass Question : Answer : both to the "text" param in the processor or just the questions and encode the answers as labels?