# Agent / tool guidance for mlx-vision

## What this repo is
Python library **mlx-vision**: DSpark-style speculative decoding for **OCR VLMs on Apple Silicon (MLX)**.
Counterpart to text-only [mlx-dspark](https://github.com/ARahim3/mlx-dspark). Built on [mlx-vlm](https://github.com/Blaizzy/mlx-vlm).

## When to use
- Faster local OCR on Mac (M-series) for DeepSeek-OCR-2, GLM-OCR, Unlimited-OCR
- Speculative decoding: `baseline` | `dspark` | `lookup` | `dflash` | `eagle3` | `mtp` | `auto`
- PDF page OCR via `pypdfium2` (`pip install 'mlx-vision[pdf]'`)

## Install
```bash
pip install mlx-vision
# from source:
pip install -e ".[pdf,dev]"
```

## Quick commands
```bash
mlx-vision models
mlx-vision -m deepseek-ocr-2 -i page.png --mode auto -v
mlx-vision -m deepseek-ocr-2 -i doc.pdf --page 0 --dpi 200
mlx-vision bench --image page.png --modes baseline,lookup,dspark
mlx-vision serve --port 8080
```

## Python
```python
from mlx_vision import ocr
r = ocr("page.png", model="deepseek-ocr-2", mode="auto")
```

## Important caveats (do not skip)
1. **No public OCR DSpark weights yet** — `--mode dspark` uses training-free semi-AR lookup + confidence schedule until drafters are trained (`scripts/train_drafter/`).
2. DeepSeek-OCR MLX checkpoints trigger Torch remote-code processor load; this package falls back to native MLX processors (`src/mlx_vision/load.py`) — do not “fix” by installing torch unless needed for training.
3. Unlimited-OCR uses **R-SWA**; draft block size is clamped to the sliding window.
4. Requires Apple Silicon + MLX (Metal).

## Key paths
| Path | Role |
|------|------|
| `src/mlx_vision/generate.py` | `ocr()` API |
| `src/mlx_vision/load.py` | Robust model/processor load |
| `src/mlx_vision/speculative/` | DSpark/DFlash/confidence/lookup |
| `src/mlx_vision/models/registry.py` | Presets + drafter registry |
| `src/mlx_vision/pdf.py` | PDF → PNG |
| `scripts/train_drafter/` | DeepSpec OCR drafter recipe |

## Related
- GitHub: https://github.com/will702/mlx-vision
- Hugging Face: https://huggingface.co/will702/mlx-vision
- Upstream OCR weights: `mlx-community/DeepSeek-OCR-2-*`, `mlx-community/GLM-OCR-*`, `baidu/Unlimited-OCR`