SAM 2: Optimized for AMD ROCm

SAM 2 is Meta's video object segmentation model that tracks and segments objects across video frames using promptable masks. This repository packages inference for video object segmentation using ONNX Runtime, exported and validated for AMD ROCm so it runs efficiently on AMD GPUs, CPUs, and NPUs.

This is based on the implementation of SAM 2 found here. This repository contains configurations and scripts optimized for AMD® ROCm™ platforms. You can use the sam2 AMD scripts to reproduce results or export with custom configurations. More details on model performance can be found here.

Task Overview

Task: Video object segmentation (semi-supervised VOS)

Dataset: DAVIS 2017 val (30 videos; smoke runs use EVAL_MAX_VIDEOS)

Output metrics: J (region similarity), F (boundary), J&F (mean)

Model variants: Four SAM 2 Hiera sizes are supported — tiny, small (default), base_plus, large. Set MODEL_SIZE on the make command line.

AMD ROCm Optimization

This model export has been adapted and validated for AMD Instinct™ / Radeon™ GPUs running ROCm, as well as AMD CPUs and AMD Ryzen AI NPUs. Key points:

Validated backends: ONNX Runtime across CPU (FP32/FP16/BF16/INT8), GPU (MIGraphX execution provider — FP32/FP16/BF16), and NPU (VitisAI execution provider — FP32/FP16; BF16/INT8 disabled by default).
No code changes required versus the upstream SAM 2 implementation — only environment/runtime configuration differs.
INT8 is disabled on GPU (gfx1151) due to a rocBLAS kernel gap.
benchmark-gpu-bf16 fails at runtime — MIGraphX throws a RuntimeException in the compiled bf16 kernel; fp16 and fp32 GPU paths work correctly.
profile-gpu-int8 fails during MIGraphX compilation — the MIGraphX ONNX parser does not support the resize mode used by SAM2 in the INT8 path.

Runtime	Precision	Backend	Hardware	Notes
ONNX Runtime	FP32 / FP16 / BF16 / INT8	CPU Execution Provider	AMD CPU	—
ONNX Runtime	FP32 / FP16 / BF16	MIGraphX Execution Provider	AMD Instinct™ / Radeon™ GPU (ROCm)	INT8 disabled on gfx1151 (rocBLAS gap); BF16 fails at runtime
ONNX Runtime	FP32 / FP16	VitisAI Execution Provider	AMD Ryzen AI NPU	BF16/INT8 disabled by default (`ENABLE_NPU_BF16_INT8=1` to enable)

Getting Started

For setup instructions, evaluation scripts, and custom configuration options, see the sam2 on GitHub.

Model Details

Model Type: Video object segmentation (promptable mask tracking) — three-stage ONNX pipeline (frame encoder, track-step init, track-step propagation)

Base Model: SAM 2 Hiera-small (default) — tiny, base_plus, large also supported

Model Stats:

Input image: (1, 3, 1024, 1024) float32 per-frame encoder input
GT mask (eval): (H, W) binary, frame 0 prompt only
Output mask: (H, W) binary, per-frame segmentation
Precision tested: FP32, FP16, BF16, INT8 (CPU); FP32, FP16, BF16 (GPU); FP32, FP16 (NPU)

Accuracy Pipeline

Higher J&F means predicted masks agree more closely with ground truth across the video — 100.0 would be perfect segmentation, 0.0 means no overlap. Published SAM 2 Hiera-small J&F on DAVIS 2017 is ~89.6.

Metrics Explained

Metric	Description
J	Region similarity (Jaccard index averaged over frames) — measures mask overlap quality.
F	Boundary F-measure — rewards precise mask boundaries, not just region overlap.
J&F	Mean of J and F — primary DAVIS semi-supervised VOS metric.

Accuracy Results

Partial Dataset Evaluation (DAVIS 2017 val, 5 of 30 videos) — default variant MODEL_SIZE=small; run EVAL_MAX_VIDEOS=10 make eval-onnx-* for full-val numbers:

Device	Precision	Videos	J	F	J&F
CPU	FP32	5 (partial)	88.55	91.50	90.03
CPU	FP16	5 (partial)	88.55	91.50	90.03
CPU	BF16	5 (partial)	88.55	91.50	90.03
CPU	INT8	5 (partial)	88.55	91.50	90.03
GPU	FP32	5 (partial)	87.22	90.35	88.79
GPU	FP16	5 (partial)	87.21	90.36	88.78
GPU	BF16	5 (partial)	87.15	90.27	88.71
NPU	FP32	5 (partial)	88.55	91.50	90.03
NPU	FP16	5 (partial)	88.55	91.50	90.03

These are partial DAVIS runs (5 of 30 videos); J&F is indicative only.

Dig Deeper

Want to explore the full evaluation scripts, config options, and other AMD-optimized model examples?

📂 View the full project on GitHub

The GitHub repository includes:

Setup and prerequisites for ROCm environments
Three-stage ONNX export (frame encoder, track-step init, track-step propagation) for CPU/GPU/NPU
DAVIS 2017 dataset staging and J/F/J&F evaluation pipeline
Benchmarking and reproduction instructions across CPU, GPU, and NPU

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

Mask Generation

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support