AGENTS.md

Agent / tool guidance for mlx-vision

What this repo is

Python library mlx-vision: DSpark-style speculative decoding for OCR VLMs on Apple Silicon (MLX). Counterpart to text-only mlx-dspark. Built on mlx-vlm.

When to use

• Faster local OCR on Mac (M-series) for DeepSeek-OCR-2, GLM-OCR, Unlimited-OCR
• Speculative decoding: baseline | dspark | lookup | dflash | eagle3 | mtp | auto
• PDF page OCR via pypdfium2 (pip install 'mlx-vision[pdf]')

Install

pip install mlx-vision
# from source:
pip install -e ".[pdf,dev]"

Quick commands

mlx-vision models
mlx-vision -m deepseek-ocr-2 -i page.png --mode auto -v
mlx-vision -m deepseek-ocr-2 -i doc.pdf --page 0 --dpi 200
mlx-vision bench --image page.png --modes baseline,lookup,dspark
mlx-vision serve --port 8080

Python

from mlx_vision import ocr
r = ocr("page.png", model="deepseek-ocr-2", mode="auto")

Important caveats (do not skip)

1. No public OCR DSpark weights yet — --mode dspark uses training-free semi-AR lookup + confidence schedule until drafters are trained (scripts/train_drafter/).
2. DeepSeek-OCR MLX checkpoints trigger Torch remote-code processor load; this package falls back to native MLX processors (src/mlx_vision/load.py) — do not “fix” by installing torch unless needed for training.
3. Unlimited-OCR uses R-SWA; draft block size is clamped to the sliding window.
4. Requires Apple Silicon + MLX (Metal).

Key paths

Path	Role
src/mlx_vision/generate.py	ocr() API
src/mlx_vision/load.py	Robust model/processor load
src/mlx_vision/speculative/	DSpark/DFlash/confidence/lookup
src/mlx_vision/models/registry.py	Presets + drafter registry
src/mlx_vision/pdf.py	PDF → PNG
scripts/train_drafter/	DeepSpec OCR drafter recipe

Related

• GitHub: https://github.com/will702/mlx-vision
• Hugging Face: https://huggingface.co/will702/mlx-vision
• Upstream OCR weights: mlx-community/DeepSeek-OCR-2-*, mlx-community/GLM-OCR-*, baidu/Unlimited-OCR