--- library_name: onnx license: apache-2.0 tags: - foundation - amd - rocm - image-segmentation pipeline_tag: mask-generation --- ![](https://huggingface.co/AMD-PAVS-AI/mobilesam/resolve/main/mobilesam.png) # MobileSAM: Optimized for AMD ROCm MobileSAM is a lightweight, promptable segmentation model — a distilled Segment Anything Model (SAM) that predicts a pixel-level mask for any object from a single point prompt. This repository packages inference for promptable instance segmentation using **ONNX Runtime**, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs, CPUs, and NPUs. This is based on the implementation of MobileSAM found [here](https://github.com/ChaoningZhang/MobileSAM). This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [mobilesam AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline). --- ## Task Overview **Task:** Promptable instance segmentation (point prompt → mask) **Dataset:** SA-1B subset (10 images, per-mask point-prompt annotations) **Output metrics:** mIoU, Mean Mask IoU, Median Mask IoU, IoU Std Dev, Min IoU, Max IoU --- ## AMD ROCm Optimization This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points: - Validated backends: **ONNX Runtime** across CPU (FP32), GPU (MIGraphX execution provider — FP32/FP16 for benchmarking, FP32/FP16/BF16/INT8 for profiling and evaluation), and NPU (VitisAI execution provider, auto-quantized internally). - The pipeline is a two-stage ONNX graph — a TinyViT image encoder (once per image) feeding a prompt/mask decoder (once per point). - No code changes required versus the upstream MobileSAM implementation — only environment/runtime configuration differs. - On GPU, the MIGraphX execution provider writes the TinyViT encoder output in NHWC memory order; the evaluator reinterprets it back to NCHW so GPU accuracy matches CPU. | Runtime | Precision | Backend | Hardware | Notes | |---|---|---|---|---| | ONNX Runtime | FP32 | CPU Execution Provider | AMD CPU | — | | ONNX Runtime | FP32 / FP16 (benchmark); FP32/FP16/BF16/INT8 (eval/profile) | MIGraphX Execution Provider | AMD Instinct™ / Radeon™ GPU (ROCm) | NHWC output reinterpreted to NCHW | | ONNX Runtime | Auto | VitisAI Execution Provider | AMD Ryzen AI NPU | Auto-quantized internally; quality lower than CPU/GPU | --- ## Getting Started For setup instructions, evaluation scripts, and custom configuration options, see the [mobilesam on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam). --- ## Model Details **Model Type:** Promptable instance segmentation — distilled Segment Anything Model with a TinyViT encoder **Base Model:** MobileSAM (pinned commit [`b01a9ccef3b9e10b099b544efe004d0871802c3b`](https://github.com/ChaoningZhang/MobileSAM/commit/b01a9ccef3b9e10b099b544efe004d0871802c3b)) **Model Stats:** - Two-stage pipeline: TinyViT image encoder + prompt/mask decoder - Encoder input: `(1, 3, 1024, 1024)` float32; encoder output (image_embeddings): `(1, 256, 64, 64)` float32 - Decoder input (point_coords): `(1, 1, 2)` float32; (point_labels): `(1, 1)` float32 - Decoder output (low_res_masks): `(1, 1, 256, 256)` float32; (iou_predictions): `(1, 1)` float32 - Precision tested: FP32 (CPU); FP32, FP16, BF16, INT8 (GPU); auto-quantized (NPU) --- ## Accuracy Pipeline Higher mIoU means the model's predicted masks overlap more closely with the ground-truth annotations — 1.0 would be pixel-perfect agreement, 0.0 means no overlap. In practice, values above ~0.5 for mIoU on SA-1B are considered strong for prompted segmentation. ### Metrics Explained | Metric | Description | |--------|-------------| | mIoU | Primary metric — mean Intersection over Union averaged across all images. Higher means predicted masks align closely with ground truth across the evaluation set. | | Mean Mask IoU | Average per-mask IoU across all individual point prompts — captures how well the model handles each mask independently, not just the per-image average. | | Median Mask IoU | Median per-mask IoU — robust to outliers; if this is much higher than Mean Mask IoU, a few bad masks are dragging the mean down. | | IoU Std Dev | Standard deviation of per-mask IoU scores — low std means consistent quality across all prompts; high std means some masks are much worse than others. | | Min IoU / Max IoU | Range of per-mask IoU — exposes the worst-case and best-case mask predictions. A very low Min IoU indicates the model occasionally produces completely wrong masks. | ### Accuracy Results **Full Dataset Evaluation (SA-1B subset, 10 images)** — filled from `runs/quality__.json`; run `make metrics` to refresh: | Device | Precision | mIoU | Mean Mask IoU | Median Mask IoU | IoU Std | |--------|-----------|------|---------------|-----------------|---------| | CPU | FP32 | 0.5752 | 0.5752 | 0.5711 | 0.0574 | | GPU | FP32 | 0.5752 | 0.5752 | 0.5711 | 0.0574 | | GPU | FP16 | 0.5750 | 0.5750 | 0.5681 | 0.0563 | | NPU | Auto | 0.4252 | 0.4252 | 0.4174 | 0.0753 | --- ## Dig Deeper Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples? 📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/mobilesam)** The GitHub repository includes: - Setup and prerequisites for ROCm environments - Scripts for CPU, GPU, and NPU runners (encoder + decoder benchmarking/profiling) - SA-1B dataset staging and mIoU evaluation pipeline - Benchmarking and reproduction instructions