README.md

---
license: other
license_name: sam-license
license_link: LICENSE
tags:
  - onnx
  - segmentation
  - sam
  - comics
---

# SAM 3.1 — ONNX export for ComicApp

ONNX conversion of Meta's **SAM 3.1** for use in
[ComicApp](https://comicapp.org). **No retraining, no fine-tuning** — this is a
format conversion of the authors' released weights so they can run under
onnxruntime.

## Attribution and licence

- **Original work:** SAM 3.1, Meta Platforms — <https://github.com/facebookresearch/sam3>
- **Licence:** the **SAM License** (19 November 2025). A verbatim copy ships in
  this repo as `LICENSE`, as §1(b)(i) requires.
- **Your use of these files is governed by that Agreement**, exactly as the
  original weights are. Notable terms: no use for military/warfare, nuclear,
  espionage or weapons purposes; no reverse engineering; compliance with export
  and sanctions controls.
- Source checkpoint: `sam3.1_multiplex.pt`.

## What is here

| file | precision | size | what it does |
|---|---|---|---|
| `sam31_image_encoder.onnx` | **fp16** | 910 MB | Page encoder. Runs ONCE per page; everything else is cheap against its output. |
| `sam31_prompt_side.onnx` + `.data` | fp32 | 121 MB | Concept ("find every X") prompting. |
| `sam31_prompt_box.onnx` + `.data` | fp32 | 134 MB | Box prompting — outline what a person drew a box around. |

The split is deliberate: the encoder is 94% of the bytes and nearly all of the
compute, so a page is encoded once and then prompted repeatedly for almost
nothing. That is what makes an interactive mask editor viable.

## Input contract — read this before wiring it up

**Normalisation is `mean = std = 0.5`** (i.e. `2x/255 − 1`), from the model's own
`processor_config.json`. **It is NOT ImageNet normalisation.** Getting this wrong
degrades results silently rather than failing.

- Encoder input: `image`, `float16`, `[1, 3, 1008, 1008]`, RGB, NCHW.
- Encoder output: four feature levels. **The decoder consumes the first
  THREE** — the model applies `scalp=1` and discards the last.
- Prompt-side outputs: `pred_logits [1,200,1]`, `pred_boxes [1,200,4]`
  (cxcywh, normalised), `pred_masks [1,200,288,288]`, `presence_logit [1,1]`.
- **Thresholding is not in the graph.** Score is
  `sigmoid(logit) * sigmoid(presence)`; apply your own floor and upsample the
  masks to page size. Keeping it outside means the confidence floor is a runtime
  setting rather than a property of the file.

## Why fp16 for the encoder

Measured, not assumed. On a real comic page, fp16 against fp32:

| | fp32 | fp16 |
|---|---|---|
| detections kept | 24 | **24 — same set** |
| mask IoU on survivors (min / mean) | 0.95060 / 0.99278 | **0.95060 / 0.99321** |
| mask-pixel disagreement | 0.1825% | **0.1803%** |
| size | 1819 MB | **910 MB** |

Identical decisions, half the download.

Note that at the *feature* level fp16 looks ~10× worse (relative error 0.11–0.17
vs 0.012–0.016) — but `mean|diff|` is 0.000089, so that is a handful of outliers
a max-based ratio exaggerates, and nothing downstream notices it. If you are
evaluating a quantisation, compare decisions rather than tensors.

The prompt-side graphs stay fp32: they are 13% of the bytes, so the saving is not
worth an unverified change.

## Conversion notes

The stock model does not export as-is. Six changes were needed, all
behaviour-preserving:

1. `perflib/fused.py:addmm_act` hardcodes bf16 and has no ONNX symbolic —
   replaced with `activation(F.linear(...))`, which is what it computes.
2. RoPE uses complex tensors (`torch.polar`, `view_as_complex`); ONNX has no
   complex type — rewritten in real arithmetic,
   `(a+bi)(c+di) = (ac−bd) + (ad+bc)i`.
3. The 32 complex buffers are converted to real `(…,2)` form **before** tracing;
   doing it inside the function puts an `aten::view_as_real` in the graph.
4. `decoder.py` compares stored ints against export-time SymInts —
   `torch.compiler.is_dynamo_compiling()` forced true, which is the honest answer
   while tracing.
5. `aten::_pin_memory` is not implemented by `torch.export`; pinning is a
   host-transfer hint with no effect on values, so it is made identity.
6. `concat_padded_sequences` short-circuits on zero-length operands. With an
   empty geometric prompt the geometry tensors are zero-**rowed**, and ONNX
   shape inference cannot broadcast over a zero-length dim.

Each was verified to reproduce the original path exactly before export.

⚠️ Do not "fix" (6) by seeding a dummy box. A whole-page box with a positive
label is not inert — it instructs the model to segment everything, and the file
will load, run, and quietly answer a different question.

## Verification

Both prompt graphs make **identical decisions** to the PyTorch reference on a
real page — same surviving query set, mask IoU ≥ 0.9991, <0.008% of mask pixels
differing — checked against a control (a different concept) that correctly
disagrees.