| # Agent / tool guidance for mlx-vision |
|
|
| ## What this repo is |
| Python library **mlx-vision**: DSpark-style speculative decoding for **OCR VLMs on Apple Silicon (MLX)**. |
| Counterpart to text-only [mlx-dspark](https://github.com/ARahim3/mlx-dspark). Built on [mlx-vlm](https://github.com/Blaizzy/mlx-vlm). |
|
|
| ## When to use |
| - Faster local OCR on Mac (M-series) for DeepSeek-OCR-2, GLM-OCR, Unlimited-OCR |
| - Speculative decoding: `baseline` | `dspark` | `lookup` | `dflash` | `eagle3` | `mtp` | `auto` |
| - PDF page OCR via `pypdfium2` (`pip install 'mlx-vision[pdf]'`) |
|
|
| ## Install |
| ```bash |
| pip install mlx-vision |
| # from source: |
| pip install -e ".[pdf,dev]" |
| ``` |
|
|
| ## Quick commands |
| ```bash |
| mlx-vision models |
| mlx-vision -m deepseek-ocr-2 -i page.png --mode auto -v |
| mlx-vision -m deepseek-ocr-2 -i doc.pdf --page 0 --dpi 200 |
| mlx-vision bench --image page.png --modes baseline,lookup,dspark |
| mlx-vision serve --port 8080 |
| ``` |
|
|
| ## Python |
| ```python |
| from mlx_vision import ocr |
| r = ocr("page.png", model="deepseek-ocr-2", mode="auto") |
| ``` |
|
|
| ## Important caveats (do not skip) |
| 1. **No public OCR DSpark weights yet** — `--mode dspark` uses training-free semi-AR lookup + confidence schedule until drafters are trained (`scripts/train_drafter/`). |
| 2. DeepSeek-OCR MLX checkpoints trigger Torch remote-code processor load; this package falls back to native MLX processors (`src/mlx_vision/load.py`) — do not “fix” by installing torch unless needed for training. |
| 3. Unlimited-OCR uses **R-SWA**; draft block size is clamped to the sliding window. |
| 4. Requires Apple Silicon + MLX (Metal). |
|
|
| ## Key paths |
| | Path | Role | |
| |------|------| |
| | `src/mlx_vision/generate.py` | `ocr()` API | |
| | `src/mlx_vision/load.py` | Robust model/processor load | |
| | `src/mlx_vision/speculative/` | DSpark/DFlash/confidence/lookup | |
| | `src/mlx_vision/models/registry.py` | Presets + drafter registry | |
| | `src/mlx_vision/pdf.py` | PDF → PNG | |
| | `scripts/train_drafter/` | DeepSpec OCR drafter recipe | |
|
|
| ## Related |
| - GitHub: https://github.com/will702/mlx-vision |
| - Hugging Face: https://huggingface.co/will702/mlx-vision |
| - Upstream OCR weights: `mlx-community/DeepSeek-OCR-2-*`, `mlx-community/GLM-OCR-*`, `baidu/Unlimited-OCR` |
|
|
