mlx-community

/

Mage-VL-8bit

like 1

Follow

MLX Community 15.5k

Hi 👋 flagging this here so anyone hitting the same wall has a fix.

Symptom

Following the model card's "Use" snippet verbatim on a normal Apple-Silicon MLX setup (no torch / torchvision installed — the standard case) fails on the first generate() call:

ValueError: Failed to process inputs with error:
ones_like(): argument 'input' (position 1) must be Tensor, not mlx.core.array

What's happening

The mage-vl branch of mlx-vlm ships a torch-free MageVLProcessor port — but its from_pretrained calls AutoImageProcessor.from_pretrained(use_fast=False), which requires torchvision. Without torchvision, that raises an ImportError which mlx-vlm's processor-patch dispatcher silently swallows, and dispatch falls back to the upstream remote-code MageVLProcessor (torch-based). The downstream prepare_inputs then crashes because it's torch trying to operate on mlx arrays.

This is upstream in mlx-vlm (not in this checkpoint's weights or configs) — filed as a comment on PR #1745 with a suggested fix.

Workaround (until the PR merges)

Drop this module into your project and import it before calling mlx_vlm.load:

# processor_patch.py
from __future__ import annotations
import importlib, json, threading

_LOCK = threading.Lock()
_INSTALLED = False

def _load_preprocessor_config(model_path_or_repo: str) -> dict:
    from pathlib import Path
    p = Path(model_path_or_repo)
    if p.exists() and p.is_dir():
        cp = p / "preprocessor_config.json"
        return json.loads(cp.read_text()) if cp.exists() else {}
    try:
        from huggingface_hub import hf_hub_download
        return json.loads(Path(hf_hub_download(model_path_or_repo, "preprocessor_config.json")).read_text())
    except Exception:
        return {}

_KW = ("patch_size","temporal_patch_size","merge_size","min_pixels","max_pixels",
       "do_rescale","rescale_factor","do_normalize","image_mean","image_std","do_convert_rgb")

def _install():
    from mlx_vlm.models.base import install_auto_processor_patch, load_chat_template
    from mlx_vlm.models.mage_vl.processing_mage_vl import MageVLProcessor as _Base

    class MageVLProcessorFixed(_Base):
        @classmethod
        def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
            from transformers import AutoTokenizer
            from mlx_vlm.models.qwen3_vl.processing_qwen3_vl import Qwen3VLImageProcessor
            kwargs.pop("use_fast", None); kwargs.pop("trust_remote_code", None)
            tok = AutoTokenizer.from_pretrained(pretrained_model_name_or_path, trust_remote_code=False, **kwargs)
            load_chat_template(tok, pretrained_model_name_or_path)
            cfg = _load_preprocessor_config(pretrained_model_name_or_path)
            ip = Qwen3VLImageProcessor(**{k: cfg[k] for k in _KW if k in cfg})
            return cls(image_processor=ip, tokenizer=tok)

    install_auto_processor_patch("mage_vl", MageVLProcessorFixed)

def ensure_installed():
    global _INSTALLED
    with _LOCK:
        if _INSTALLED: return
        importlib.import_module("mlx_vlm.models.mage_vl")  # forces the buggy patch first
        _install()
        _INSTALLED = True

ensure_installed()

It layers a corrected MageVLProcessor on top of mlx-vlm's dispatch chain by building a numpy/PIL-only Qwen3VLImageProcessor directly (the same image processor mlx-vlm's qwen3_vl port uses — identical schema to Qwen2VLImageProcessor, no torch/torchvision). Importing this module before mlx_vlm.load ensures AutoProcessor.from_pretrained("mlx-community/Mage-VL-8bit") returns the torch-free MageVLProcessor instead of falling through to the torch remote-code one.

Verified end-to-end: generate(model, processor, prompt, image=["dog.jpg"]) on the COCO cats image returns "Two cats are sleeping on a red couch with remote controls next to them."

Suggestion for the model card

It'd help future users to either (a) add a one-line note that the mage-vl branch port has an upstream issue with torchvision-free setups + link to PR #1745, or (b) pin a working mlx-vlm revision here once the fix lands. The current card sends users directly into the broken path.

Thanks for the quantization work — the int8 numbers in the card (5.0 GB resident, 88.3 tok/s decode on M5 Max) match what I'm seeing once this is patched.