File size: 2,816 Bytes
82cef12 c39d802 82cef12 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 | ---
license: apache-2.0
base_model: microsoft/Mage-VL
tags:
- mlx
- apple-silicon
- vision-language
- image-text-to-text
- video
- streaming
pipeline_tag: image-text-to-text
library_name: mlx
---
# Mage-VL 4-bit (MLX, Apple Silicon)
An **MLX** (4-bit) conversion of [microsoft/Mage-VL](https://huggingface.co/microsoft/Mage-VL) —
a 5B image/video vision-language model (Qwen3-4B text backbone + a from-scratch
**Mage-ViT** "Codec-ViT" vision encoder) — that runs on Apple Silicon.
Port code, converter, and validators: **https://github.com/rsravanreddy/Mage-VL-MLX**
- Weights: `3.1 GB` (4-bit, group size 64)
- Runs image, video (frame-sampling), and the **streaming event gate** locally on a Mac.
## Usage
```bash
# 1. install the MLX stack + register the mage_vl plugin
pip install mlx mlx-lm mlx-vlm numpy pillow tokenizers jinja2 av
git clone https://github.com/rsravanreddy/Mage-VL-MLX && cd Mage-VL-MLX
ln -s "$PWD/mage_vl" "$(python -c 'import mlx_vlm,os;print(os.path.dirname(mlx_vlm.__file__))')/models/mage_vl"
# 2. download these weights
hf download sr29/Mage-VL-mlx-4bit --local-dir mage-vl-mlx
# 3. run (image or video)
python scripts/generate.py --mlx mage-vl-mlx --tokenizer-src mage-vl-mlx \
--image path/to/image.jpg --prompt "Describe this image."
python scripts/generate.py --mlx mage-vl-mlx --tokenizer-src mage-vl-mlx \
--video path/to/video.mp4 --num-frames 8 --prompt "What is happening?"
```
## Performance (Apple M4, 16GB)
| model | weights | image decode | image peak RAM |
|-------|--------:|-------------:|---------------:|
| 4-bit | 3.1 GB | 30.6 tok/s | 4.65 GB |
| 8-bit | 5.0 GB | 19.1 tok/s | 6.55 GB |
4-bit is recommended for 16GB; 8-bit gives richer output if you have RAM.
## Validation
- **Image preprocessing**: bit-exact vs the HF `Qwen2VLImageProcessor` (max_abs_diff 0.0).
- **Vision tower**: numerically matches the reference weights end-to-end
(`max_abs_diff 3.0e-4`, fp32, full 24 layers).
- **Streaming Mamba mixer**: matches a canonical selective-scan reference (`4.3e-7`).
- **Generation**: qualitatively correct on image + video.
- **Not yet done**: full end-to-end logit parity vs the HF model (memory-gated);
the codec (token-reduction) video backend needs the external codec engine.
## Streaming
Mage-VL's proactive **event gate** (`streammind_gate`) is ported
(`mage_vl/streaming.py`). See `scripts/stream.py` for a per-frame silent/speak
timeline. Note: the gate weights (`streammind_gate.safetensors`) are separate and
downloaded from the upstream Mage-VL repo.
## License & attribution
Apache-2.0. Derivative of [microsoft/Mage-VL](https://huggingface.co/microsoft/Mage-VL)
(Apache-2.0); reuses the Qwen3 language model from
[mlx-vlm](https://github.com/Blaizzy/mlx-vlm). Weights converted, not retrained.
|