Azaz666/Qwen2.5-VL-3B-Instruct-GPTQ-Int4

Qwen__Qwen2.5-VL-3B-Instruct__gptq_int4

This is a GPTQ (4-bit) quantized version of Qwen/Qwen2.5-VL-3B-Instruct.

Quantization Details

• Method: GPTQ
• Bits: 4
• Base model: Qwen/Qwen2.5-VL-3B-Instruct
• Group size: 128
• Quantized portion: LLM backbone only
• Vision tower: FP16 (unchanged)
• Loading: AutoModelForImageTextToText / AutoModelForCausalLM (trust_remote_code where applicable)

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "{REPO_ID}",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("{REPO_ID}", trust_remote_code=True)

Replace {REPO_ID} with the repo ID of this model.

Original Model

See Qwen/Qwen2.5-VL-3B-Instruct for the original FP16 model.