Azaz666/SmolVLM-256M-Instruct-HQQ-INT4-v2

HuggingFaceTB__SmolVLM-256M-Instruct__hqq_int4

This is a HQQ (4-bit) quantized version of HuggingFaceTB/SmolVLM-256M-Instruct.

Quantization Details

• Method: HQQ
• Bits: 4
• Base model: HuggingFaceTB/SmolVLM-256M-Instruct
• Group size: 64
• FP16 modules: vision_model, vision_tower, visual, connector, multi_modal_projector, mlp1, lm_head, embed_tokens, patch_embed
• Calibration: none (HQQ is calibration-free)
• Backend: hqq.core.quantize.HQQLinear

Usage

from transformers import AutoProcessor, AutoModelForImageTextToText
import torch

model = AutoModelForImageTextToText.from_pretrained(
    "{REPO_ID}",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True,
)
processor = AutoProcessor.from_pretrained("{REPO_ID}", trust_remote_code=True)

Replace {REPO_ID} with the repo ID of this model.

Original Model

See HuggingFaceTB/SmolVLM-256M-Instruct for the original FP16 model.