| license: mit | |
| library_name: mlx | |
| tags: | |
| - mlx | |
| - florence2 | |
| - omniparser | |
| - ui | |
| - apple-silicon | |
| base_model: microsoft/OmniParser-v2.0 | |
| pipeline_tag: image-text-to-text | |
| # OmniParser icon_caption — MLX | |
| **MLX (bfloat16) conversion of [microsoft/OmniParser-v2.0](https://huggingface.co/microsoft/OmniParser-v2.0)'s `icon_caption`** — a Florence-2 fine-tuned on UI elements, for captioning interactive icons in screenshots. Runs on Apple Silicon via `mlx_vlm` with **no PyTorch**. | |
| License: **MIT** (© Microsoft Corporation) — see `LICENSE`. This repo redistributes the original MIT-licensed weights converted to MLX format. | |
| ## Usage | |
| Needs a small no-torch patch on transformers 5.x (register `florence2_language`, route the image processor to `CLIPImageProcessorPil`). See the conversion recipe at the bottom. | |
| ```python | |
| # apply the florence2 no-torch patch first (see recipe), then: | |
| from mlx_vlm import load, generate | |
| model, processor = load("PlusMinus1/omniparser-icon-caption-mlx") | |
| out = generate(model, processor, "<CAPTION>", image=["icon_crop.png"], max_tokens=20) | |
| ``` | |
| ## Provenance | |
| - Base: microsoft/Florence-2-base (MIT) | |
| - Fine-tune: microsoft/OmniParser-v2.0 `icon_caption` (MIT) | |
| - Conversion: `mlx_vlm.convert` (bfloat16) + a transformers-5.x no-torch compatibility patch. | |