README.md

---
license: other
license_name: apache-2.0
license_link: https://huggingface.co/facebook/sam3/blob/main/LICENSE
base_model:
- facebook/sam3
library_name: mlx
tags:
- mlx
- apple-silicon
- segmentation
- sam3
- image-segmentation
- vision
language:
- en
pipeline_tag: image-segmentation
---

# 🎯 MLX SAM3

**Segment Anything Model 3 — Native Apple Silicon Implementation**

<p align="center">
  <a href="https://github.com/ml-explore/mlx"><img src="https://img.shields.io/badge/MLX-Framework-blue" alt="MLX"></a>
  <a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/Python-3.13+-green" alt="Python 3.13+"></a>
  <a href="https://github.com/Deekshith-Dade/mlx_sam3"><img src="https://img.shields.io/badge/GitHub-Repository-black" alt="GitHub"></a>
</p>

This is an **MLX port** of [Meta's SAM3](https://huggingface.co/facebook/sam3) model, optimized for native execution on Apple Silicon (M1/M2/M3/M4) Macs.

> 📖 **Learn more**: Check out the [accompanying blog post](https://deekshith.me/blog/mlx-sam3) explaining the SAM3 architecture and this implementation.

## Model Description

SAM3 (Segment Anything Model 3) is a powerful image segmentation model that can segment objects in images using:
- **Text prompts** — Describe what you want to segment ("car", "person", "dog")
- **Box prompts** — Draw bounding boxes to include or exclude regions

This MLX port provides native Apple Silicon performance, leveraging Apple's MLX framework for optimized inference on Mac.

## Intended Uses

- **Interactive image segmentation** on Apple Silicon Macs
- **Object detection and masking** with text descriptions
- **Region-based segmentation** using bounding box prompts
- **Rapid prototyping** of segmentation workflows on Mac

## How to Use

### Installation

```bash
# Clone the repository
git clone https://github.com/Deekshith-Dade/mlx_sam3.git
cd mlx-sam3

# Install with uv (recommended)
uv sync

# Or with pip
pip install -e .
```

### Python API

```python
from PIL import Image
from sam3 import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor

# Load model (auto-downloads weights on first run)
model = build_sam3_image_model()
processor = Sam3Processor(model, confidence_threshold=0.5)

# Load and process an image
image = Image.open("your_image.jpg")
state = processor.set_image(image)

# Segment with text prompt
state = processor.set_text_prompt("person", state)

# Access results
masks = state["masks"]       # Binary segmentation masks
boxes = state["boxes"]       # Bounding boxes [x0, y0, x1, y1]
scores = state["scores"]     # Confidence scores

print(f"Found {len(scores)} objects")
```

### Web Interface

Launch the interactive web application:

```bash
cd app && ./run.sh
```

- **Frontend**: http://localhost:3000
- **API**: http://localhost:8000
- **API Docs**: http://localhost:8000/docs

## Requirements

| Requirement | Version | Notes |
|-------------|---------|-------|
| **macOS** | 13.0+ | Apple Silicon required (M1/M2/M3/M4) |
| **Python** | 3.13+ | Required for MLX compatibility |
| **Node.js** | 18+ | For the web interface (optional) |

> ⚠️ **Apple Silicon Only**: This implementation uses MLX, which is optimized exclusively for Apple Silicon.

## Model Details

- **Architecture**: SAM3 with ViTDet backbone
- **Framework**: MLX (Apple's machine learning framework)
- **Weights**: Converted from original PyTorch weights
- **Model Size**: ~3.5GB

## Limitations

- Runs **only on Apple Silicon** Macs (M1/M2/M3/M4)
- Requires macOS 13.0 or later
- Python 3.13+ required for MLX compatibility

## Citation

If you use this model, please cite the original SAM3 paper and this MLX implementation:

```bibtex
@misc{mlx-sam3,
  author = {Deekshith Dade},
  title = {MLX SAM3: Native Apple Silicon Implementation},
  year = {2024},
  url = {https://github.com/Deekshith-Dade/mlx_sam3}
}
```

## Links

- **GitHub Repository**: [https://github.com/Deekshith-Dade/mlx_sam3](https://github.com/Deekshith-Dade/mlx_sam3)
- **Blog Post**: [https://deekshith.me/blog/mlx-sam3](https://deekshith.me/blog/mlx-sam3)
- **Original SAM3**: [https://huggingface.co/facebook/sam3](https://huggingface.co/facebook/sam3)

## Acknowledgments

- [Meta AI](https://ai.meta.com/) for the original SAM3 model
- [Apple MLX Team](https://github.com/ml-explore/mlx) for the MLX framework
- The open-source community for continuous inspiration

---

**Built with ❤️ for Apple Silicon**