In the sample code there is a comment that says tune encoder and lora:

tune vision encoder and lora

model.set_lora_adapter('vision')
for param in model.model.embed_tokens_extend.image_embed.parameters():
    param.requires_grad = True

Does it mean we are finetuning both vision encoder and Lora adapters for vision?