OmniParser icon_caption — MLX
MLX (bfloat16) conversion of microsoft/OmniParser-v2.0's icon_caption — a Florence-2 fine-tuned on UI elements, for captioning interactive icons in screenshots. Runs on Apple Silicon via mlx_vlm with no PyTorch.
License: MIT (© Microsoft Corporation) — see LICENSE. This repo redistributes the original MIT-licensed weights converted to MLX format.
Usage
Needs a small no-torch patch on transformers 5.x (register florence2_language, route the image processor to CLIPImageProcessorPil). See the conversion recipe at the bottom.
# apply the florence2 no-torch patch first (see recipe), then:
from mlx_vlm import load, generate
model, processor = load("PlusMinus1/omniparser-icon-caption-mlx")
out = generate(model, processor, "<CAPTION>", image=["icon_crop.png"], max_tokens=20)
Provenance
- Base: microsoft/Florence-2-base (MIT)
- Fine-tune: microsoft/OmniParser-v2.0
icon_caption(MIT) - Conversion:
mlx_vlm.convert(bfloat16) + a transformers-5.x no-torch compatibility patch.
- Downloads last month
- 51
Model size
0.3B params
Tensor type
BF16
·
Hardware compatibility
Log In to add your hardware
Quantized
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for PlusMinus1/omniparser-icon-caption-mlx
Base model
microsoft/OmniParser-v2.0
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("PlusMinus1/omniparser-icon-caption-mlx") config = load_config("PlusMinus1/omniparser-icon-caption-mlx") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output)