BiRefNet general as a Core AI .aimodel (float16), with README and licence

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+cutout-birefnet-float16.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 ZhengPeng
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

NOTICE

@@ -0,0 +1,11 @@
+cutout-birefnet: BiRefNet general (ZhengPeng7/BiRefNet at revision
+e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4), MIT, Copyright (c) 2024 ZhengPeng.
+See LICENSE.
+
+Changed for Core AI, in export_birefnet.py:
+- torchvision's deform_conv2d is computed from gathers and matrix products
+  (identical result: max |alpha difference| 4.6e-6 in float32 on the CPU).
+- The ImageNet normalisation and the final sigmoid are inside the graph; only
+  the last, full-size prediction is returned.
+- Weights and arithmetic are float16; the input and output stay float32.
+The trained weights are otherwise the authors', unmodified.

README.md

@@ -0,0 +1,107 @@
+---
+license: mit
+base_model: ZhengPeng7/BiRefNet
+tags:
+  - background-removal
+  - image-matting
+  - dichotomous-image-segmentation
+  - birefnet
+  - core-ai
+  - apple-silicon
+  - macos
+library_name: swift-object-cutout
+pipeline_tag: image-segmentation
+---
+
+# cutout-birefnet: BiRefNet general for Core AI (macOS 27)
+
+[BiRefNet](https://github.com/ZhengPeng7/BiRefNet) general — Zheng, Gao, Fan, Liu, Laaksonen,
+Ouyang and Sebe, *Bilateral Reference for High-Resolution Dichotomous Image Segmentation*, CAAI
+Artificial Intelligence Research, 2024; code and weights MIT — exported to an Apple Core AI
+`.aimodel`. Give it a photo and it returns a soft alpha matte of the main subject: a product, a
+plant, a glass, an animal, a person — the background-removal job, on device.
+
+It is what [`cutout`](https://github.com/arraypress/swift-cutout-cli) uses for `--subject` and
+`--person`, through [swift-object-cutout](https://github.com/arraypress/swift-object-cutout), and
+what `design` uses to lift a subject off its photo. Both download this repository by themselves the
+first time it is needed, pinned to one commit, every file checked against its published hash.
+Apple's built-in Vision matte stays as the no-download fallback.
+
+## Files
+
+| Path | What |
+|---|---|
+| `cutout-birefnet-float16.aimodel/` | One entry point, `main`: `image` `[1,3,1024,1024]` float32, RGB in 0…1 → `alpha` `[1,1,1024,1024]` float32 in 0…1. Float16 weights and arithmetic, 463 MB. |
+| `export_birefnet.py` | The export script (`uv run export_birefnet.py --dtype float16`). |
+| `LICENSE`, `NOTICE` | The authors' MIT licence, and what the export changed. |
+
+The host resizes the picture to 1024×1024 without keeping its aspect ratio (as the network was
+trained) and stretches the alpha back to the picture's size with bilinear sampling. ImageNet
+normalisation and the sigmoid are inside the asset.
+
+## What changed
+
+Core AI has no deformable convolution, which BiRefNet's decoder uses. The export computes it from
+what it is: for each kernel tap, a bilinear sample of the feature map (four gathers from a
+zero-padded copy), scaled by the modulator, then a matrix product with that tap's weights, summed
+over the taps. Sample positions stay float32 in the float16 export. Against the authors' code in
+float32 on the CPU, the rewrite differs by at most 4.6 × 10⁻⁶ in alpha.
+
+## Fidelity
+
+Run through Core AI on the GPU, against the authors' PyTorch model (float32, CPU) on the same 30
+inputs at 1024²:
+
+| Export | Mean abs. alpha diff. | 99.9th percentile | Worst pixel | Mask IoU (>0.5), worst image |
+|---|---|---|---|---|
+| float32 | 1.9 × 10⁻⁷ | 2.4 × 10⁻⁵ | 1.3 × 10⁻⁴ | 0.99996 |
+| float16 (this repo) | 1.9 × 10⁻⁵ | 1.5 × 10⁻³ | 0.068 | 0.9985 |
+
+End to end (Swift, Core Graphics resizing) the float16 asset scores the same as PyTorch on the
+true-alpha sets below: SAD 23.40 against 23.39 on objects and 6.12 against 6.10 on animals.
+
+## Quality
+
+Measured with the standard matting metrics against true alpha on AIM-500 (Li et al., 2021), which
+this checkpoint was not trained on (its training sets are DIS5K, DUTS, HRSOD, UHRSD, HRS10K, P3M-10k
+and a humans set). 100 objects (plants, furniture, toys, fruit, glass), 100 animals and the 100
+portraits, at their own size (at most 1920 px). Lower is better except IoU; the edge band is the
+uncertain strip round the subject where hair and fur live.
+
+| Set | Matte | SAD | MSE ×10³ | Grad | Conn | IoU | Edge-band MSE ×10³ |
+|---|---|---|---|---|---|---|---|
+| Objects | Apple Vision subject mask, cutout's crisp edge | 81.39 | 33.09 | 0.690 | 82.82 | 0.802 | 92.5 |
+| Objects | **this model** | **23.40** | **3.29** | **0.211** | **22.87** | **0.919** | **12.6** |
+| Animals | Apple Vision subject mask, crisp edge | 17.17 | 5.79 | 0.309 | 17.15 | 0.976 | 47.7 |
+| Animals | **this model** | **6.12** | **0.99** | **0.074** | **5.50** | **0.991** | **9.4** |
+| Portraits | Apple Vision person segmentation, crisp edge | 24.91 | 10.63 | 0.224 | 24.45 | 0.962 | 36.0 |
+| Portraits | **this model** | **4.86** | **0.62** | **0.058** | **4.19** | **0.992** | **6.7** |
+
+It beat Vision by SAD on 99 of the 100 objects, all 100 animals and 99 of the 100 portraits. Vision found no subject at all in 4 of the objects (counted
+as empty mattes); glass is the hardest category for both (SAD 211 for Vision, 51 here). Re-solving this
+model's alpha from colour (the crisp edge that helps Vision's softer mask) made every set worse, so
+cutout passes this model's alpha through.
+
+## Speed and memory
+
+Apple M3 Max, macOS 27, GPU: the first load compiles the model for the Mac in about 15 s and keeps a
+553 MB compiled copy in `~/Library/Caches/coreai-cache`; later loads are quick. About 0.9 s per
+1024² inference from Python, and 0.9–1.3 s per photo through `cutout` end to end (decode, model, edge,
+PNG). A folder run of 100 photos peaked at 3.2–3.8 GB of memory, against 5.6 GB for the float32 export
+and up to 10 GB for PyTorch on MPS.
+
+## Use
+
+```sh
+cutout photo.jpg                 # fetches this model on first use
+cutout ./shots --out ./cut       # a folder, the model loaded once
+cutout model remove subject      # the model and its compiled copy
+```
+
+Or by hand: `hf download arraypress/cutout-birefnet --local-dir models`, then point
+`$CUTOUT_SUBJECT_MODEL` at the `.aimodel`. Requires macOS 27 (Core AI) on Apple silicon. Converted
+with coreai-torch 0.4.3, coreai-core 1.0.0b3, torch 2.9.0 and transformers 4.57.1.
+
+## Licence
+
+MIT, as the original: see `LICENSE` (the authors' copyright) and `NOTICE` (what the export changed).

cutout-birefnet-float16.aimodel/main.hash

@@ -0,0 +1 @@
+j�@��پō��4s`]!��tU��S/�w��fw

cutout-birefnet-float16.aimodel/main.mlirb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6aad408d80d9bec58df1e93473605d21169dfe7455bda3532f987708add66677
+size 463261242

cutout-birefnet-float16.aimodel/metadata.json

@@ -0,0 +1,8 @@
+{
+  "producer" : "coreai-core 1.0.0b3",
+  "creationDate" : "20261016T175354Z",
+  "description" : "BiRefNet general (ZhengPeng7\/BiRefNet@e2bf8e4), float16. Entry 'main': image [1,3,1024,1024] float32 RGB 0…1 → alpha [1,1,1024,1024] float32 0…1. Resize to 1024² and back in the host.",
+  "assetVersion" : "2.0",
+  "license" : "MIT",
+  "author" : "Zheng, Gao, Fan, Liu, Laaksonen, Ouyang, Sebe — BiRefNet (CAAI AIR 2024), general; Core AI export by ObjectCutout"
+}

export_birefnet.py

@@ -0,0 +1,140 @@
+# /// script
+# requires-python = ">=3.11,<3.13"
+# dependencies = ["torch==2.9.0", "torchvision==0.24.0", "transformers==4.57.1", "timm", "kornia", "einops",
+#                 "numpy", "pillow", "coreai-torch==0.4.3", "coreai-core==1.0.0b3"]
+# ///
+"""BiRefNet general (Zheng et al., CAAI AIR 2024; MIT) → cutout-birefnet-<dtype>.aimodel.
+
+    uv run Tools/export_birefnet.py [--dtype float16|float32] [--out Tools/exports] [--check image.jpg …]
+
+The network is the authors' own (ZhengPeng7/BiRefNet at a pinned revision, loaded with its remote code); one
+change, made for the runtime and checked against the original: torchvision's `deform_conv2d`, which Core AI does
+not have, is rewritten as what it computes — each kernel tap's bilinear sample (four gathers from the zero-padded
+feature map), scaled by the modulator, then a matrix product with that tap's weights, summed over the taps.
+Sample positions are computed in float32 even in a float16 export: at 256 px a half float steps by 0.125, which
+would move every bilinear weight.
+
+One entry point, `main`: `image` [1,3,1024,1024] float32 RGB in 0…1 (resized by the host) → `alpha`
+[1,1,1024,1024] float32 in 0…1. The ImageNet normalisation and the final sigmoid are inside. In a float16 export
+the weights and arithmetic are half; input and output stay float32, so the host is the same for both.
+
+--check runs the original network (torchvision's deform_conv2d) and the rewritten one on each image in float32 on
+the CPU and prints the alpha difference, before anything is exported."""
+import argparse, os, sys, time, shutil
+from pathlib import Path
+import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
+
+HERE = Path(__file__).resolve().parent
+REPOSITORY = "ZhengPeng7/BiRefNet"
+REVISION = "e2bf8e4460fc8fa32bba5ea4d94b3233d367b0e4"
+SIZE = 1024
+
+ap = argparse.ArgumentParser()
+ap.add_argument("--dtype", default="float16", choices=["float16", "float32"])
+ap.add_argument("--out", default=str(HERE / "exports"))
+ap.add_argument("--check", nargs="*", default=[], help="images to compare original vs rewritten on (float32, CPU)")
+ap.add_argument("--no-export", action="store_true")
+args = ap.parse_args()
+
+from transformers import AutoModelForImageSegmentation
+torch.set_num_threads(4)
+model = AutoModelForImageSegmentation.from_pretrained(REPOSITORY, trust_remote_code=True, revision=REVISION).float().eval()
+remote = sys.modules[type(model).__module__]
+print(f"BiRefNet general @ {REVISION[:7]}: {sum(p.numel() for p in model.parameters())/1e6:.1f} M parameters", flush=True)
+
+
+def deform_conv2d(x, offset, weight, bias, padding, mask):
+    """torchvision.ops.deform_conv2d (stride 1, dilation 1, one offset group), from gathers and matmuls."""
+    _, C, H, W = x.shape
+    O, _, kh, kw = weight.shape
+    pad = padding if isinstance(padding, int) else padding[0]
+    Ho, Wo = H + 2 * pad - kh + 1, W + 2 * pad - kw + 1
+    Hp, Wp = H + 2, W + 2
+    # One row of zeros round the map: every corner that falls outside reads 0, as torchvision's does.
+    table = F.pad(x, (1, 1, 1, 1)).reshape(C, Hp * Wp).t()                      # [Hp*Wp, C]
+    off = offset.float().reshape(kh * kw, 2, Ho, Wo)                              # (dy, dx) per tap
+    mod = mask.float().reshape(kh * kw, Ho * Wo, 1)
+    ys = torch.arange(Ho, dtype=torch.float32).reshape(Ho, 1) - pad
+    xs = torch.arange(Wo, dtype=torch.float32).reshape(1, Wo) - pad
+    out = None
+    for k in range(kh * kw):
+        ky, kx = divmod(k, kw)
+        py = ys + ky + off[k, 0]
+        px = xs + kx + off[k, 1]
+        y0, x0 = torch.floor(py), torch.floor(px)
+        ly, lx = py - y0, px - x0
+        y0c, y1c = y0.clamp(-1, H) + 1, (y0 + 1).clamp(-1, H) + 1
+        x0c, x1c = x0.clamp(-1, W) + 1, (x0 + 1).clamp(-1, W) + 1
+        sample = None
+        for yc, xc, w in ((y0c, x0c, (1 - ly) * (1 - lx)), (y0c, x1c, (1 - ly) * lx),
+                          (y1c, x0c, ly * (1 - lx)), (y1c, x1c, ly * lx)):
+            index = (yc * Wp + xc).reshape(-1).to(torch.int64)
+            term = F.embedding(index, table) * (w.reshape(-1, 1) * mod[k]).to(x.dtype)
+            sample = term if sample is None else sample + term
+        part = sample @ weight[:, :, ky, kx].t()                                  # [Ho*Wo, O]
+        out = part if out is None else out + part
+    out = out.t().reshape(1, O, Ho, Wo)
+    return out if bias is None else out + bias.reshape(1, O, 1, 1)
+
+
+def rewritten_forward(self, x):
+    offset = self.offset_conv(x)
+    modulator = 2. * torch.sigmoid(self.modulator_conv(x))
+    return deform_conv2d(x, offset, self.regular_conv.weight, self.regular_conv.bias, self.padding, modulator)
+
+
+original_forward = remote.DeformableConv2d.forward
+MEAN = torch.tensor([0.485, 0.456, 0.406]).reshape(1, 3, 1, 1)
+STD = torch.tensor([0.229, 0.224, 0.225]).reshape(1, 3, 1, 1)
+
+
+class Net(nn.Module):
+    """0…1 RGB in, alpha out: the normalisation and the sigmoid inside, the last (full-size) prediction only."""
+    def __init__(self, m, dtype):
+        super().__init__(); self.m = m.to(dtype); self.dtype = dtype
+        self.register_buffer("mean", MEAN.to(dtype)); self.register_buffer("std", STD.to(dtype))
+    def forward(self, image):
+        x = (image.to(self.dtype) - self.mean) / self.std
+        return self.m(x)[-1].sigmoid().to(torch.float32)
+
+
+if args.check:
+    from PIL import Image
+    with torch.no_grad():
+        for path in args.check:
+            im = Image.open(path).convert("RGB").resize((SIZE, SIZE), Image.BILINEAR)
+            x = torch.from_numpy(np.asarray(im, np.float32) / 255).permute(2, 0, 1)[None]
+            remote.DeformableConv2d.forward = original_forward
+            a = Net(model, torch.float32)(x)
+            remote.DeformableConv2d.forward = rewritten_forward
+            b = Net(model, torch.float32)(x)
+            d = (a - b).abs()
+            print(f"{Path(path).name}: |original − rewritten| max {d.max():.2e} mean {d.mean():.2e}", flush=True)
+
+if args.no_export:
+    sys.exit(0)
+
+import coreai_torch
+from coreai.runtime import AIModelAssetMetadata
+remote.DeformableConv2d.forward = rewritten_forward
+dtype = torch.float16 if args.dtype == "float16" else torch.float32
+net = Net(model, dtype).eval()
+t0 = time.time()
+converter = coreai_torch.TorchConverter(mode=coreai_torch.TorchConverter.Mode.RELEASE)
+with torch.no_grad():
+    ex = torch.export.export(net, (torch.rand(1, 3, SIZE, SIZE),)).run_decompositions(coreai_torch.get_decomp_table())
+    print(f"traced in {time.time()-t0:.0f} s", flush=True)
+    converter.add_exported_program(ex, input_names=["image"], output_names=["alpha"], entrypoint_name="main")
+program = converter.to_coreai()
+print(f"converted in {time.time()-t0:.0f} s", flush=True)
+meta = AIModelAssetMetadata()
+meta.author = "Zheng, Gao, Fan, Liu, Laaksonen, Ouyang, Sebe — BiRefNet (CAAI AIR 2024), general; Core AI export by ObjectCutout"
+meta.license = "MIT"
+meta.model_description = (f"BiRefNet general ({REPOSITORY}@{REVISION[:7]}), {args.dtype}. Entry 'main': image [1,3,{SIZE},{SIZE}] "
+                          f"float32 RGB 0…1 → alpha [1,1,{SIZE},{SIZE}] float32 0…1. Resize to {SIZE}² and back in the host.")
+meta.creation_date = int(time.time())
+out = Path(args.out); out.mkdir(parents=True, exist_ok=True)
+asset = out / f"cutout-birefnet-{args.dtype}.aimodel"
+if asset.exists(): shutil.rmtree(asset)
+program.save_asset(asset, meta)
+print(f"saved {asset} ({sum(f.stat().st_size for f in asset.rglob('*'))/1e6:.0f} MB) in {time.time()-t0:.0f} s", flush=True)