README.md

---
library_name: onnx
license: apache-2.0
tags:
  - foundation
  - amd
  - rocm
  - image-segmentation
pipeline_tag: mask-generation
---

![](https://huggingface.co/AMD-PAVS-AI/mobilesam/resolve/main/mobilesam.png)

# MobileSAM: Optimized for AMD ROCm

MobileSAM is a lightweight, promptable segmentation model — a distilled Segment Anything Model (SAM) that predicts a pixel-level mask for any object from a single point prompt. This repository packages inference for promptable instance segmentation using **ONNX Runtime**, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs, CPUs, and NPUs.

This is based on the implementation of MobileSAM found [here](https://github.com/ChaoningZhang/MobileSAM).
This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [mobilesam AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline).

---

## Task Overview

**Task:** Promptable instance segmentation (point prompt → mask)

**Dataset:** SA-1B subset (10 images, per-mask point-prompt annotations)

**Output metrics:** mIoU, Mean Mask IoU, Median Mask IoU, IoU Std Dev, Min IoU, Max IoU

---

## AMD ROCm Optimization

This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points:

- Validated backends: **ONNX Runtime** across CPU (FP32), GPU (MIGraphX execution provider — FP32/FP16 for benchmarking, FP32/FP16/BF16/INT8 for profiling and evaluation), and NPU (VitisAI execution provider, auto-quantized internally).
- The pipeline is a two-stage ONNX graph — a TinyViT image encoder (once per image) feeding a prompt/mask decoder (once per point).
- No code changes required versus the upstream MobileSAM implementation — only environment/runtime configuration differs.
- On GPU, the MIGraphX execution provider writes the TinyViT encoder output in NHWC memory order; the evaluator reinterprets it back to NCHW so GPU accuracy matches CPU.

| Runtime | Precision | Backend | Hardware | Notes |
|---|---|---|---|---|
| ONNX Runtime | FP32 | CPU Execution Provider | AMD CPU | — |
| ONNX Runtime | FP32 / FP16 (benchmark); FP32/FP16/BF16/INT8 (eval/profile) | MIGraphX Execution Provider | AMD Instinct™ / Radeon™ GPU (ROCm) | NHWC output reinterpreted to NCHW |
| ONNX Runtime | Auto | VitisAI Execution Provider | AMD Ryzen AI NPU | Auto-quantized internally; quality lower than CPU/GPU |

---

## Getting Started

For setup instructions, evaluation scripts, and custom configuration options, see the [mobilesam on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam).

---

## Model Details

**Model Type:** Promptable instance segmentation — distilled Segment Anything Model with a TinyViT encoder

**Base Model:** MobileSAM (pinned commit [`b01a9ccef3b9e10b099b544efe004d0871802c3b`](https://github.com/ChaoningZhang/MobileSAM/commit/b01a9ccef3b9e10b099b544efe004d0871802c3b))

**Model Stats:**
- Two-stage pipeline: TinyViT image encoder + prompt/mask decoder
- Encoder input: `(1, 3, 1024, 1024)` float32; encoder output (image_embeddings): `(1, 256, 64, 64)` float32
- Decoder input (point_coords): `(1, 1, 2)` float32; (point_labels): `(1, 1)` float32
- Decoder output (low_res_masks): `(1, 1, 256, 256)` float32; (iou_predictions): `(1, 1)` float32
- Precision tested: FP32 (CPU); FP32, FP16, BF16, INT8 (GPU); auto-quantized (NPU)

---

## Accuracy Pipeline

Higher mIoU means the model's predicted masks overlap more closely with the ground-truth annotations — 1.0 would be pixel-perfect agreement, 0.0 means no overlap. In practice, values above ~0.5 for mIoU on SA-1B are considered strong for prompted segmentation.

### Metrics Explained

| Metric | Description |
|--------|-------------|
| mIoU | Primary metric — mean Intersection over Union averaged across all images. Higher means predicted masks align closely with ground truth across the evaluation set. |
| Mean Mask IoU | Average per-mask IoU across all individual point prompts — captures how well the model handles each mask independently, not just the per-image average. |
| Median Mask IoU | Median per-mask IoU — robust to outliers; if this is much higher than Mean Mask IoU, a few bad masks are dragging the mean down. |
| IoU Std Dev | Standard deviation of per-mask IoU scores — low std means consistent quality across all prompts; high std means some masks are much worse than others. |
| Min IoU / Max IoU | Range of per-mask IoU — exposes the worst-case and best-case mask predictions. A very low Min IoU indicates the model occasionally produces completely wrong masks. |

### Accuracy Results

**Full Dataset Evaluation (SA-1B subset, 10 images)** — filled from `runs/quality_<device>_<precision>.json`; run `make metrics` to refresh:

<!-- accuracy-table-start -->
| Device | Precision | mIoU | Mean Mask IoU | Median Mask IoU | IoU Std |
|--------|-----------|------|---------------|-----------------|---------|
| CPU | FP32 | 0.5752 | 0.5752 | 0.5711 | 0.0574 |
| GPU | FP32 | 0.5752 | 0.5752 | 0.5711 | 0.0574 |
| GPU | FP16 | 0.5750 | 0.5750 | 0.5681 | 0.0563 |
| NPU | Auto | 0.4252 | 0.4252 | 0.4174 | 0.0753 |
<!-- accuracy-table-end -->

---

## Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam)**

The GitHub repository includes:
- Setup and prerequisites for ROCm environments
- Scripts for CPU, GPU, and NPU runners (encoder + decoder benchmarking/profiling)
- SA-1B dataset staging and mIoU evaluation pipeline
- Benchmarking and reproduction instructions