allenai

/

Molmo2-4B

like 46

Follow

Ai2 5.73k

Package:

• transformers: 4.57.6
• llmcompressor: 0.10.0.1
• compressed-tensors: 0.14.0.1

Script:

from transformers import AutoProcessor,  AutoModelForImageTextToText

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "allenai/Molmo2-4B"

# Load model.
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)

# Configure the quantization algorithm and scheme.
# In this case, we:
#   * quantize the weights to fp8 with channel-wise quantization
#   * quantize the activations to fp8 with dynamic token activations
recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_DYNAMIC",
    ignore=[
        "re:.*lm_head",
    ],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Save to disk in compressed-tensors format.
SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-DYNAMIC"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)

Error message:

Traceback (most recent call last):
  File "/home/ubuntu/projects/eval_vlm/convert_to_fp8.py", line 33, in <module>
    processor.save_pretrained(SAVE_DIR)
  File "/home/ubuntu/projects/eval_vlm/.venv/lib/python3.12/site-packages/transformers/processing_utils.py", line 860, in save_pretrained
    if self.audio_tokenizer is not None:
       ^^^^^^^^^^^^^^^^^^^^
AttributeError: 'Molmo2Processor' object has no attribute 'audio_tokenizer'