Qwen2-VL-7B-checkpoint

This is a LoRA checkpoint for fine-tuning the Qwen2-VL-7B-Instruct model. The checkpoint is contained inside the qwen2_7B folder.

Usage

To use this LoRA with the base model, load the base model and apply the LoRA checkpoint like this:

from peft import PeftModel
from transformers import Qwen2VLForConditionalGeneration

# Define the model type and path to the fine-tuned LoRA adapter
model_path = "Qwen/Qwen2-VL-7B"
path_to_adapter = "path_to_lora_checkpoint"

# Load the base pre-trained model
model = Qwen2VLForConditionalGeneration.from_pretrained(
    model_path, torch_dtype='auto', device_map='cpu', attn_implementation='flash_attention_2'
)

# Load the LoRA adapter and move the model to GPU
lora_model = PeftModel.from_pretrained(
    model,
    path_to_adapter,
    device_map="auto",  # Automatically allocate model layers to available devices
    trust_remote_code=True
).eval().cuda()

# Your code here