| --- |
| license: other |
| license_name: apache-2.0 |
| license_link: https://huggingface.co/facebook/sam3/blob/main/LICENSE |
| base_model: |
| - facebook/sam3 |
| library_name: mlx |
| tags: |
| - mlx |
| - apple-silicon |
| - segmentation |
| - sam3 |
| - image-segmentation |
| - vision |
| language: |
| - en |
| pipeline_tag: image-segmentation |
| --- |
| |
| # 🎯 MLX SAM3 |
|
|
| **Segment Anything Model 3 — Native Apple Silicon Implementation** |
|
|
| <p align="center"> |
| <a href="https://github.com/ml-explore/mlx"><img src="https://img.shields.io/badge/MLX-Framework-blue" alt="MLX"></a> |
| <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.13+-green" alt="Python 3.13+"></a> |
| <a href="https://github.com/Deekshith-Dade/mlx_sam3"><img src="https://img.shields.io/badge/GitHub-Repository-black" alt="GitHub"></a> |
| </p> |
|
|
| This is an **MLX port** of [Meta's SAM3](https://huggingface.co/facebook/sam3) model, optimized for native execution on Apple Silicon (M1/M2/M3/M4) Macs. |
|
|
| > 📖 **Learn more**: Check out the [accompanying blog post](https://deekshith.me/blog/mlx-sam3) explaining the SAM3 architecture and this implementation. |
|
|
| ## Model Description |
|
|
| SAM3 (Segment Anything Model 3) is a powerful image segmentation model that can segment objects in images using: |
| - **Text prompts** — Describe what you want to segment ("car", "person", "dog") |
| - **Box prompts** — Draw bounding boxes to include or exclude regions |
|
|
| This MLX port provides native Apple Silicon performance, leveraging Apple's MLX framework for optimized inference on Mac. |
|
|
| ## Intended Uses |
|
|
| - **Interactive image segmentation** on Apple Silicon Macs |
| - **Object detection and masking** with text descriptions |
| - **Region-based segmentation** using bounding box prompts |
| - **Rapid prototyping** of segmentation workflows on Mac |
|
|
| ## How to Use |
|
|
| ### Installation |
|
|
| ```bash |
| # Clone the repository |
| git clone https://github.com/Deekshith-Dade/mlx_sam3.git |
| cd mlx-sam3 |
| |
| # Install with uv (recommended) |
| uv sync |
| |
| # Or with pip |
| pip install -e . |
| ``` |
|
|
| ### Python API |
|
|
| ```python |
| from PIL import Image |
| from sam3 import build_sam3_image_model |
| from sam3.model.sam3_image_processor import Sam3Processor |
| |
| # Load model (auto-downloads weights on first run) |
| model = build_sam3_image_model() |
| processor = Sam3Processor(model, confidence_threshold=0.5) |
| |
| # Load and process an image |
| image = Image.open("your_image.jpg") |
| state = processor.set_image(image) |
| |
| # Segment with text prompt |
| state = processor.set_text_prompt("person", state) |
| |
| # Access results |
| masks = state["masks"] # Binary segmentation masks |
| boxes = state["boxes"] # Bounding boxes [x0, y0, x1, y1] |
| scores = state["scores"] # Confidence scores |
| |
| print(f"Found {len(scores)} objects") |
| ``` |
|
|
| ### Web Interface |
|
|
| Launch the interactive web application: |
|
|
| ```bash |
| cd app && ./run.sh |
| ``` |
|
|
| - **Frontend**: http://localhost:3000 |
| - **API**: http://localhost:8000 |
| - **API Docs**: http://localhost:8000/docs |
|
|
| ## Requirements |
|
|
| | Requirement | Version | Notes | |
| |-------------|---------|-------| |
| | **macOS** | 13.0+ | Apple Silicon required (M1/M2/M3/M4) | |
| | **Python** | 3.13+ | Required for MLX compatibility | |
| | **Node.js** | 18+ | For the web interface (optional) | |
|
|
| > ⚠️ **Apple Silicon Only**: This implementation uses MLX, which is optimized exclusively for Apple Silicon. |
|
|
| ## Model Details |
|
|
| - **Architecture**: SAM3 with ViTDet backbone |
| - **Framework**: MLX (Apple's machine learning framework) |
| - **Weights**: Converted from original PyTorch weights |
| - **Model Size**: ~3.5GB |
|
|
| ## Limitations |
|
|
| - Runs **only on Apple Silicon** Macs (M1/M2/M3/M4) |
| - Requires macOS 13.0 or later |
| - Python 3.13+ required for MLX compatibility |
|
|
| ## Citation |
|
|
| If you use this model, please cite the original SAM3 paper and this MLX implementation: |
|
|
| ```bibtex |
| @misc{mlx-sam3, |
| author = {Deekshith Dade}, |
| title = {MLX SAM3: Native Apple Silicon Implementation}, |
| year = {2024}, |
| url = {https://github.com/Deekshith-Dade/mlx_sam3} |
| } |
| ``` |
|
|
| ## Links |
|
|
| - **GitHub Repository**: [https://github.com/Deekshith-Dade/mlx_sam3](https://github.com/Deekshith-Dade/mlx_sam3) |
| - **Blog Post**: [https://deekshith.me/blog/mlx-sam3](https://deekshith.me/blog/mlx-sam3) |
| - **Original SAM3**: [https://huggingface.co/facebook/sam3](https://huggingface.co/facebook/sam3) |
|
|
| ## Acknowledgments |
|
|
| - [Meta AI](https://ai.meta.com/) for the original SAM3 model |
| - [Apple MLX Team](https://github.com/ml-explore/mlx) for the MLX framework |
| - The open-source community for continuous inspiration |
|
|
| --- |
|
|
| **Built with ❤️ for Apple Silicon** |
|
|
|
|
