MarcinEU/finegrain-box-segmenter-ONNX

finegrain-box-segmenter — ONNX

FP32 ONNX export of finegrain/finegrain-box-segmenter (an MVANet high-resolution background remover), ready to run in the browser, Node.js and Python with ONNX Runtime (CPU, WebGPU (recommended), DirectML).

• Base model: finegrain/finegrain-box-segmenter (MIT) — MVANet, SafeTensors, arXiv:2404.07445
• Architecture: MVANet with a Swin-B backbone, ~94.6 M parameters; static 1024×1024 input, batch fixed at 1.
• What this repo adds: the same network exported to ONNX (full-precision floating-point fp32), so it runs without PyTorch/refiners.
• Why fp32: maximum compatibility with zero precision risk — the fp32 graph runs on CPU, WebGPU and DirectML, and stays bit-identical to the original release; at ~94.6 M parameters it is still cheap enough to run on a local machine. (An fp16 build (~403 MB) exists and its masks matched fp32 in testing, but it is not published, to keep the release simple — open a discussion if you need it.) Regarding why this release uses the "quantized" tag: it's a fairly common convention for ONNX / format derivatives (same as e.g. onnx-community/BiRefNet-ONNX), so I decided to go this route for tagging.
• Output is bit-identical to the PyTorch reference on CPU (mask MAE 0.000 / 255; random-input logits max|Δ| = 1.5e-5).

Examples

These tests are intentionally made to be difficult for the models, doing their best to expose the weak points of each model.

1. Sub-pixel hair strands
2. Refraction + small object touching frame edge
3. Large shape touching edge in a style rarely seen in the training data (impasto), no obvious dominant subject
4. 3D/miniature with glows and object blending under water
5. Cartoon / flat 2D
6. Complex fire pattern
7. Reflections on a confusing background
8. Intricate many-holed topology

The short version: finegrain-box-segmenter is strongest on soft, semi-transparent edges — it keeps more sub-pixel hair strands (1), cleaner glass/refraction edges (2) and more of the flame and fur tips (6) than the other tools, and it resolves the many-holed branch topology (8) cleanly. Its weak spots in this set: the ultra-thin rigging wires in (4) (the BiRefNet variants preserve them better) and the unusual impasto style of (3), where part of the rooster's comb goes semi-transparent. All cutouts were produced the same way: each tool's standard whole-image pipeline — no boxes or prompts — on identical source images, and every crop is the same pixel window. The competitors (finegrain is this model): ben2 = BEN2, bria = BRIA RMBG-2.0, birefnet = BiRefNet, the massive-trained checkpoint (BiRefNet-massive-TR_DIS5K_TR_TEs-epoch_420, run via rembg).

	Original	finegrain	ben2	bria	birefnet
1
2
3
4
5
6
7
8

Files

File	Precision	Size	Notes
onnx/model.onnx	fp32	~805 MB	full precision; CPU + WebGPU; bit-identical parity with PyTorch

ONNX operator set version 17, ~94.6 M parameters, SHA256(onnx/model.onnx) = d3f60fc778f2b14435a8df3d61511c293cec3a88c670ad8ecac5035c6b9730d5

Why 805 MB for a 94.6 M-param model: ~378 MB is fp32 weights; the other ~425 MB is constant-folded shifted-window (Swin) attention masks / position embeddings baked in at export (do_constant_folding=True). Note that disabling the folding will make the file larger, not smaller.

I/O contract

	name	shape	dtype
input	input	[1, 3, 1024, 1024]	float32
output	logits	[1, 1, 1024, 1024]	float32 (raw logits)

Pre-processing (must be reproduced by the caller): take the RGB image, resize to 1024×1024 (bilinear or a comparable resampler — mild resampler differences don't visibly change the mask), scale to [0,1] (/255), normalize with ImageNet statistics mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225], layout NCHW.

Post-processing: apply sigmoid to the logits to get an alpha matte in [0,1], then resize it back to the original image size. Use it directly as a mask, or as the alpha channel of an RGBA cutout. (Note: this model emits raw logits — apply sigmoid; do not min-max normalize the output the way some other ONNX matting models, e.g. BEN2, require.)

How it was converted

Exported from the published v0.1 SafeTensors weights with PyTorch's legacy TorchScript exporter (torch.onnx.export with the dynamo path disabled) at opset 17, with constant folding enabled. The raw nn.Module — obtained from refiners' BoxSegmenter and put in eval / float mode — is traced on a single 1×3×1024×1024 float input, and its tensors are named input and logits. There are no architecture changes and no retraining: the exported graph is exactly image → logits.

Two non-obvious choices were required:

• Legacy exporter, not dynamo. torch's dynamo / torch.export path fails decomposing Swin's transpose(1,2).reshape(...) (it lowers the non-contiguous view to a strict aten.view that can't represent the transposed strides). The legacy TorchScript tracer records a real Reshape and exports cleanly.
• adaptive_avg_pool2d → avg_pool2d. Refiners' pooling always divides evenly (adaptive_avg_pool2d(x, (h//r, w//r))), so it is numerically identical to a plain avg_pool2d(kernel=r, stride=r) that maps to ONNX AveragePool — an exact swap, not an approximation.

The graph uses only standard ops (Conv, AveragePool, MatMul, Gemm, Resize, Softmax, LayerNormalization, PRelu, Add, Mul, Concat, Split, Slice, Reshape, Transpose) — no grid_sample / einsum / unfold — so it runs on ORT CPU and WebGPU as-is.

Parity (verify_parity.py, CPU fp32): random input → torch vs ORT logits max|Δ| = 1.5e-5 (sigmoid 3e-6); refiners' golden cactus image → mask MAE = 0.000 / 255 (bit-identical). The full sourced write-up lives in this repo at docs/HOW-CONVERSION-WAS-MADE.md; to reproduce the export yourself, see docs/DEVELOPMENT.md and the python/ scripts.

Quality — measured on this ONNX

The base model's published benchmark was re-run on this ONNX export: the full finegrain/finegrain-product-masks-lite set (120 masks), scored with PySODMetrics. It was scored in two modes, which measure different things — don't mix them up when comparing models:

Mode (same 120 masks)	MAE ↓	S-measure ↑	E-measure (mean) ↑	Dice (mean) ↑
Box-prompted crop — the base model's published protocol	0.0079	0.9738	0.9854	0.9669
Whole-image (no box) — how this release is used	0.086	0.796	0.793	0.712

Box-prompted crop mode confirms the conversion: bit-identical parity implies identical task quality, and running the base model's own protocol on this ONNX reproduces its published numbers to within rounding:

Metric	This ONNX	Published (base)	Δ
MAE ↓	0.0079	0.0078	+0.0001
S-measure ↑	0.9738	0.974	−0.0002
E-measure (mean) ↑	0.9854	0.985	+0.0004
Dice (mean) ↑	0.9669	0.967	−0.0001

n = 120. The sub-0.001 deltas are host-side resize / box-derivation rounding, not conversion error — the ONNX inherits the original's quality (which, on this set, beats briaai/RMBG-1.4 and box-guided ZhengPeng7/BiRefNet; see the base card).

Whole-image mode (no box) scores far lower by construction: this dataset's ground truth is one specific boxed object, so scoring the no-prompt salient output against it penalizes the model whenever the image's salient region isn't exactly that boxed target. It's a worst-case lower bound on a box-oriented set — not representative of ordinary single-subject background removal (see the Examples above for what whole-image output actually looks like).

Usage — Transformers.js (easiest)

This repo ships the standard onnx/model.onnx + config.json + preprocessor_config.json layout, so Transformers.js drives the whole pipeline for you — pre-process, sigmoid, and resize-back included:

import { pipeline } from '@huggingface/transformers';

const segmenter = await pipeline('background-removal', 'MarcinEU/finegrain-box-segmenter-ONNX');
const out = await segmenter(['https://example.com/photo.jpg']);
out[0].save('cutout.png');     // RawImage RGBA - alpha is the matte
// out[0].toCanvas() / await out[0].toBlob() if you'd rather not save to disk

Verified end-to-end (@huggingface/transformers v4.2.0): the model resolves to SwinForSemanticSegmentation, Transformers.js remaps the processor output onto our input tensor and auto-applies sigmoid to the raw logits — no custom code. Add { device: 'webgpu' } for the GPU. Use v4.2.0 or newer (the automatic input-name remap and auto-sigmoid are required). If you need full control over pre/post-processing, use the hand-rolled paths below.

Usage — onnxruntime-web (browser, WebGPU)

import * as ort from 'onnxruntime-web/webgpu';

const SIZE = 1024, MEAN = [0.485,0.456,0.406], STD = [0.229,0.224,0.225];
const session = await ort.InferenceSession.create(
  'https://huggingface.co/MarcinEU/finegrain-box-segmenter-ONNX/resolve/main/onnx/model.onnx',
  { executionProviders: ['webgpu'] });

// draw your image into a 1024x1024 canvas, then:
const { data } = ctx.getImageData(0, 0, SIZE, SIZE);      // RGBA, Uint8ClampedArray
const plane = SIZE * SIZE, chw = new Float32Array(3 * plane);
for (let p = 0; p < plane; p++) {
  chw[p]             = (data[p*4]   / 255 - MEAN[0]) / STD[0];
  chw[plane + p]     = (data[p*4+1] / 255 - MEAN[1]) / STD[1];
  chw[2*plane + p]   = (data[p*4+2] / 255 - MEAN[2]) / STD[2];
}
const out = await session.run({ input: new ort.Tensor('float32', chw, [1,3,SIZE,SIZE]) });
const logits = out.logits.data;                            // Float32Array, 1024*1024
const alpha = logits.map(v => 1 / (1 + Math.exp(-v)));     // sigmoid -> [0,1]
// resize `alpha` back to your image size and composite as the alpha channel.

Usage — onnxruntime-node (Node.js, with sharp)

A complete, runnable background remover is in usage/remove_bg.mjs — run it from the repo root (with onnx/model.onnx downloaded):

npm i onnxruntime-node sharp
node usage/remove_bg.mjs --image photo.jpg --model onnx/model.onnx               # CPU
node usage/remove_bg.mjs --image photo.jpg --model onnx/model.onnx --ep webgpu   # GPU (often 3-10x faster; see Performance below)

It resizes the whole image to 1024², normalizes, runs the session, applies sigmoid, resizes the mask back, and writes *_mask.png and *_cutout.png (RGBA). Outputs carry real transparency — view them over a checkerboard/solid background.

Usage — Python (onnxruntime)

Verified against this exact export (pip install onnxruntime pillow numpy huggingface_hub); masks match the Node reference pipeline to ~1/255 mean difference:

from huggingface_hub import hf_hub_download
from PIL import Image
import numpy as np, onnxruntime as ort

model_path = hf_hub_download("MarcinEU/finegrain-box-segmenter-ONNX", "onnx/model.onnx")
session = ort.InferenceSession(model_path)

img = Image.open("photo.jpg").convert("RGB")
x = np.asarray(img.resize((1024, 1024), Image.BILINEAR), dtype=np.float32) / 255.0
x = ((x - [0.485, 0.456, 0.406]) / [0.229, 0.224, 0.225]).transpose(2, 0, 1)[None].astype(np.float32)

logits = session.run(None, {"input": x})[0][0, 0]
alpha = 1.0 / (1.0 + np.exp(-logits))                       # sigmoid -> [0,1]
mask = Image.fromarray((alpha * 255).astype(np.uint8)).resize(img.size, Image.BILINEAR)

cutout = img.copy(); cutout.putalpha(mask)
cutout.save("cutout.png"); mask.save("mask.png")

Performance across environments

The graph is a static 1024×1024 fp32 Swin-B (MVANet) running at internal batch-5, so it is compute-bound and the inference time is ~constant regardless of input image size — the source resolution only changes the negligible host-side resize (pre + post, ≈30–500 ms for 1080p→8K). The numbers below are therefore "per image" for any resolution. All EPs run the same fp32 model (no fp16/NPU path). Measured with onnxruntime-node 1.26, 15 runs/cell, median.

Measured — median inference time, seconds per image (lower is better)

Machine (form factor)	CPU (cores/threads)	GPU	RAM	CPU EP	WebGPU	DirectML
Desktop + discrete GPU	i7-8700K (6C/12T)	NVIDIA RTX 5070 Ti 16 GB	32 GB	9.7	0.9 ⭐	1.5
Laptop (Arrow Lake-H)	Core Ultra 7 255H (16C/16T)	Intel Arc 140T iGPU	48 GB	4.6	8.6	1.6 ⭐
Laptop (Tiger Lake)	i7-1165G7 (4C/8T)	Intel Iris Xe 96 EU iGPU	16 GB	13.9	9.6 ⭐	✗ OOM at init
Laptop (Raptor Lake-U)	i5-1335U (10C/12T, 15 W)	Intel Iris Xe 80 EU iGPU	8 GB	≈15 ⭐	✗ OOM ~3934 MB¹	✗ device-hung

⭐ = fastest working backend on that machine. ✗ = execution provider failed (footnotes ¹–³ below give the byte-level cause). Numbers here are the steady-state median across sizes; the per-size mean ± std is in the Reference anchors tables below. Add pre+post for end-to-end total: ≈0.05 s at 512²/1080p, ≈0.5 s at 8K.

Neither the NPU nor any other accelerator is involved. onnxruntime-node has no NPU EP here: the CPU EP uses CPU cores, DirectML/WebGPU use the GPU. The Core Ultra 7 255H's "Intel AI Boost" NPU is not used — its NPU TOPS are irrelevant to these numbers; the iGPU figures above are the Arc 140T graphics, not the NPU.

Reference anchors — match your own machine

Every hardware × EP combination measured, broken out by input size. Cells are mean ± std of the per-image inference time in seconds (the warm-up run is dropped, so n ≈ 14; n = 10 where noted). ✗ = the EP failed — see footnotes. Because inference is ~size-independent (static 1024² input) a row's four cells are nearly equal; the size axis mainly exposes where memory runs out (e.g. the 8 GB laptop clears 8K but not the smaller sizes). fp16 TFLOPS are a published proxy for ranking GPUs (the model runs fp32, so real rates are lower); discrete-GPU bandwidth is on-card VRAM, iGPU bandwidth is the shared-RAM memcpy rate measured. ″ = same as the row above; the Arc 140T additionally has XMX matrix engines (74 INT8 TOPS) that its fp32 path does not use.

GPU backends (anchor = fp16 throughput + memory bandwidth):

GPU (arch)	EP	fp16 (proxy)	Mem BW	8K (s)	4K (s)	1080p (s)	512² (s)
RTX 5070 Ti (Blackwell), discrete 16 GB	WebGPU	88 TFLOPS	896 GB/s	0.95 ± 0.07	0.85 ± 0.06	0.85 ± 0.07	0.89 ± 0.08
RTX 5070 Ti	DirectML	″	″	1.50 ± 0.03	1.50 ± 0.06	1.48 ± 0.04	1.50 ± 0.04
Arc 140T iGPU (Xe-LPG, 8 Xe @2.25 GHz)	DirectML	~9 TFLOPS	~39 GB/s sh.	1.58 ± 0.02	1.57 ± 0.00	1.57 ± 0.00	1.57 ± 0.00
Arc 140T	WebGPU	″	″	8.61 ± 0.05	8.64 ± 0.08	8.64 ± 0.09	8.65 ± 0.07
Iris Xe 96 EU iGPU (Tiger Lake)	WebGPU	3.4 TFLOPS	~31 GB/s sh.	9.79 ± 0.40	9.52 ± 0.34	9.46 ± 0.25	9.77 ± 0.46
Iris Xe 96 EU	DirectML	″	″	✗²	✗²	✗²	✗²
Iris Xe 80 EU iGPU (Raptor Lake-U)	WebGPU	~3 TFLOPS	~29 GB/s sh.	33.4 ± 2.8¹	✗¹	✗¹	✗¹
Iris Xe 80 EU	DirectML	″	″	✗³	✗³	✗³	✗³

¹ WebGPU / Iris Xe 80 EU (8 GB RAM): peaked at ~3934 MB — essentially the entire ~3940 MB shared budget (≈50% of 8 GB RAM). Only 8K ran (11 of 15 runs, ~33 s mean) before it OOM'd; every 4K / 1080p / 512 run failed. WebGPU is unusable at 8 GB RAM.
² DirectML / Iris Xe 96 EU (16 GB RAM): out of memory (8007000E) during session creation (~12.7 s in) — never ran. DirectML's footprint is ~16–19 GB (peak where it ran) vs only ~8032 MB shared iGPU budget (≈50% of 16 GB RAM).
³ DirectML / Iris Xe 80 EU (8 GB RAM): init failed with a GPU device-hung error (887A0007, "GPU will not respond to more commands") — not strictly OOM, likely a cascade from the preceding WebGPU crash on the same ~3940 MB shared budget.

CPU backend (anchor = cores × clock × AVX width + RAM bandwidth; EP = CPU on every row):

CPU (cores/threads, arch)	RAM BW	8K (s)	4K (s)	1080p (s)	512² (s)
Core Ultra 7 255H (16C/16T, Arrow Lake-H, AVX2)	~39 GB/s	4.57 ± 0.08	4.61 ± 0.05	4.63 ± 0.12	4.65 ± 0.17
i7-8700K (6C/12T, Coffee Lake, AVX2)	~25 GB/s	9.75 ± 0.24	9.69 ± 0.10	9.67 ± 0.11	9.67 ± 0.13
i7-1165G7 (4C/8T, Tiger Lake, AVX2/512)	~31 GB/s	14.6 ± 1.5	13.9 ± 0.5	13.8 ± 0.3	13.9 ± 0.2
i5-1335U (10C/12T, 15 W, Raptor Lake-U)	~29 GB/s	14.3 ± 2.1	14.9 ± 1.6	15.7 ± 2.0	14.3 ± 1.8

(The CPU EP was also run on the RTX 5070 Ti box — that's the i7-8700K row — since the CPU EP ignores the GPU. The 1335U's larger ± is real run-to-run jitter on a 15 W thermally-throttled laptop.)

How to read it / rules of thumb:

• Choose the EP, not just the chip. Discrete NVIDIA/AMD → WebGPU (fastest). Intel Arc iGPU → DirectML (its WebGPU path is ~5× slower here). Older Intel Iris Xe → WebGPU if you have ≥16 GB RAM, else CPU. No usable GPU → CPU (scales with cores × clock × AVX width; the 16-core 255H is ~3× the 4-core 1165G7).
• RAM is the gate for iGPUs. The fp32 model is ~805 MB on disk but a GPU session peaks at 8–20 GB (VRAM on discrete cards; shared system RAM on iGPUs — WebGPU peaked ~8 GB, DirectML ~15–19 GB in our runs). So: iGPU WebGPU needs ≥16 GB system RAM; iGPU DirectML wants ≥32 GB; 8 GB machines can only use the CPU. Discrete GPUs need ≥8 GB VRAM free.
• DirectML is hit-or-miss. It gave the best iGPU time (Arc 140T, 1.6 s) but failed to initialize on three of four machines (driver/VRAM dependent). Always have the CPU EP as a fallback.
• Predict GPU time ≈ scale the RTX 5070 Ti's 0.9 s by its fp16 proxy vs yours, then add ~30–50 % headroom because the model is fp32 and won't fully use tensor cores. (e.g. a ~40 TFLOPS-fp16 card → expect ~2 s; a ~9 TFLOPS iGPU via a good runtime → ~1.5–2 s, which matches the Arc 140T on DML.)

Notes

• Resolution / batch are fixed at 1×3×1024×1024 (the network hard-codes 1024² internally). Any aspect ratio works — the squash-resize matches the original model's behavior (a very common approach among mask-generation and image-segmentation models).
• Runtime requirement: the graph is opset 17, so it needs onnxruntime-node ≥ 1.12.0 (July 2022; earlier versions will crash on load). Current onnxruntime-web is fine.
• WebGPU: runs as-is on standard adapters — the widest Concat/Split is only 4–5 inputs, well under the maxStorageBuffersPerShaderStage limit, so no graph surgery is needed (verified on the machines in the Performance section above). Only WebGPU "compatibility mode" (limit 4) devices would need a cascade rewrite.

License & attribution

MIT, inherited from the base model. All credit for the weights and architecture goes to Finegrain (finegrain/finegrain-box-segmenter) and the MVANet authors. This repository only provides an ONNX conversion of the public v0.1 weights (model.safetensors, SHA256 fd5f13919dfc0dda102df1af648c3773c61221aa65fe58d6af978637baded1ae).

Citation

@article{mvanet,
  title   = {Multi-view Aggregation Network for Dichotomous Image Segmentation},
  author  = {Yu, Qian and Zhao, Xiaoqi and Pang, Youwei and Zhang, Lihe and Lu, Huchuan},
  journal = {CVPR},
  year    = {2024}
}