| --- |
| library_name: onnx |
| license: apache-2.0 |
| tags: |
| - foundation |
| - amd |
| - rocm |
| - video-segmentation |
| pipeline_tag: mask-generation |
| --- |
| |
|  |
|
|
| # SAM 2: Optimized for AMD ROCm |
|
|
| SAM 2 is Meta's video object segmentation model that tracks and segments objects across video frames using promptable masks. This repository packages inference for video object segmentation using **ONNX Runtime**, exported and validated for **AMD ROCm** so it runs efficiently on AMD GPUs, CPUs, and NPUs. |
|
|
| This is based on the implementation of SAM 2 found [here](https://github.com/facebookresearch/sam2). |
| This repository contains configurations and scripts optimized for **AMD® ROCm™** platforms. You can use the [sam2 AMD scripts](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/sam2) to reproduce results or export with custom configurations. More details on model performance can be found [here](#accuracy-pipeline). |
|
|
| --- |
|
|
| ## Task Overview |
|
|
| **Task:** Video object segmentation (semi-supervised VOS) |
|
|
| **Dataset:** DAVIS 2017 val (30 videos; smoke runs use `EVAL_MAX_VIDEOS`) |
|
|
| **Output metrics:** J (region similarity), F (boundary), J&F (mean) |
|
|
| > **Model variants:** Four SAM 2 Hiera sizes are supported — `tiny`, `small` (default), `base_plus`, `large`. Set `MODEL_SIZE` on the make command line. |
|
|
| --- |
|
|
| ## AMD ROCm Optimization |
|
|
| This model export has been adapted and validated for **AMD Instinct™ / Radeon™ GPUs** running **ROCm**, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points: |
|
|
| - Validated backends: **ONNX Runtime** across CPU (FP32/FP16/BF16/INT8), GPU (MIGraphX execution provider — FP32/FP16/BF16), and NPU (VitisAI execution provider — FP32/FP16; BF16/INT8 disabled by default). |
| - No code changes required versus the upstream SAM 2 implementation — only environment/runtime configuration differs. |
| - INT8 is disabled on GPU (gfx1151) due to a rocBLAS kernel gap. |
| - `benchmark-gpu-bf16` fails at runtime — MIGraphX throws a `RuntimeException` in the compiled bf16 kernel; fp16 and fp32 GPU paths work correctly. |
| - `profile-gpu-int8` fails during MIGraphX compilation — the MIGraphX ONNX parser does not support the resize mode used by SAM2 in the INT8 path. |
|
|
| | Runtime | Precision | Backend | Hardware | Notes | |
| |---|---|---|---|---| |
| | ONNX Runtime | FP32 / FP16 / BF16 / INT8 | CPU Execution Provider | AMD CPU | — | |
| | ONNX Runtime | FP32 / FP16 / BF16 | MIGraphX Execution Provider | AMD Instinct™ / Radeon™ GPU (ROCm) | INT8 disabled on gfx1151 (rocBLAS gap); BF16 fails at runtime | |
| | ONNX Runtime | FP32 / FP16 | VitisAI Execution Provider | AMD Ryzen AI NPU | BF16/INT8 disabled by default (`ENABLE_NPU_BF16_INT8=1` to enable) | |
|
|
| --- |
|
|
| ## Getting Started |
|
|
| For setup instructions, evaluation scripts, and custom configuration options, see the [sam2 on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/sam2). |
|
|
| --- |
|
|
| ## Model Details |
|
|
| **Model Type:** Video object segmentation (promptable mask tracking) — three-stage ONNX pipeline (frame encoder, track-step init, track-step propagation) |
|
|
| **Base Model:** SAM 2 Hiera-small (default) — `tiny`, `base_plus`, `large` also supported |
|
|
| **Model Stats:** |
| - Input image: `(1, 3, 1024, 1024)` float32 per-frame encoder input |
| - GT mask (eval): `(H, W)` binary, frame 0 prompt only |
| - Output mask: `(H, W)` binary, per-frame segmentation |
| - Precision tested: FP32, FP16, BF16, INT8 (CPU); FP32, FP16, BF16 (GPU); FP32, FP16 (NPU) |
|
|
| --- |
|
|
| ## Accuracy Pipeline |
|
|
| Higher J&F means predicted masks agree more closely with ground truth across the video — 100.0 would be perfect segmentation, 0.0 means no overlap. Published SAM 2 Hiera-small J&F on DAVIS 2017 is ~89.6. |
|
|
| ### Metrics Explained |
|
|
| | Metric | Description | |
| |--------|-------------| |
| | J | Region similarity (Jaccard index averaged over frames) — measures mask overlap quality. | |
| | F | Boundary F-measure — rewards precise mask boundaries, not just region overlap. | |
| | J&F | Mean of J and F — primary DAVIS semi-supervised VOS metric. | |
|
|
| ### Accuracy Results |
|
|
| **Partial Dataset Evaluation (DAVIS 2017 val, 5 of 30 videos)** — default variant `MODEL_SIZE=small`; run `EVAL_MAX_VIDEOS=10 make eval-onnx-*` for full-val numbers: |
|
|
| <!-- accuracy-table-start --> |
| | Device | Precision | Videos | J | F | J&F | |
| |--------|-----------|--------|-----|-----|------| |
| | CPU | FP32 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| | CPU | FP16 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| | CPU | BF16 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| | CPU | INT8 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| | GPU | FP32 | 5 (partial) | 87.22 | 90.35 | 88.79 | |
| | GPU | FP16 | 5 (partial) | 87.21 | 90.36 | 88.78 | |
| | GPU | BF16 | 5 (partial) | 87.15 | 90.27 | 88.71 | |
| | NPU | FP32 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| | NPU | FP16 | 5 (partial) | 88.55 | 91.50 | 90.03 | |
| <!-- accuracy-table-end --> |
|
|
| These are partial DAVIS runs (5 of 30 videos); J&F is indicative only. |
|
|
| --- |
|
|
| ## Dig Deeper |
|
|
| Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples? |
|
|
| 📂 **[View the full project on GitHub](https://github.com/AMD-PAVS/physical_ai_sdk/blob/main/models/sam2)** |
|
|
| The GitHub repository includes: |
| - Setup and prerequisites for ROCm environments |
| - Three-stage ONNX export (frame encoder, track-step init, track-step propagation) for CPU/GPU/NPU |
| - DAVIS 2017 dataset staging and J/F/J&F evaluation pipeline |
| - Benchmarking and reproduction instructions across CPU, GPU, and NPU |
|
|
