# ECSeg-S / ECSeg-M ONNX optimization & quantization report **Question.** Should AnnotateIt replace the shipped FP32 ECSeg-S / ECSeg-M instance-segmentation checkpoints with an optimized (graph-optimized / FP16 / INT8) variant — judged on size **and** speed **and** quality **and** stability **and** browser compatibility together, not size alone? **Decision: keep FP32 for both models (details in [§8](#8-verdicts)).** | Model | Verdict | Best format | Size before → after | Browser warm latency before → after | Quality vs FP32 | |---|---|---|---|---|---| | **ECSeg-S** | **REJECT** | FP32 (unchanged) | 39.95 MiB → 39.95 MiB (no adoptable smaller variant) | 1299 ms → 1299 ms (every smaller variant is slower or won't run) | unchanged (candidates lose quality or produce 0 instances) | | **ECSeg-M** | **REJECT** | FP32 (unchanged) | 77.08 MiB → 77.08 MiB | 2093 ms → 2093 ms | unchanged | Nothing was changed in the app or in the checkpoint registry — there is no winning variant. The reusable conversion + browser-benchmark tooling and this report are the application-repository deliverables. > Reproduce everything: [`scripts/model-optimization/README.md`](../../scripts/model-optimization/README.md). > Generated `.onnx` variants are **not** committed (large, rebuildable). > Machine-readable results: [`data/sizes.csv`](data/sizes.csv), [`data/correctness.csv`](data/correctness.csv), > [`data/latency.csv`](data/latency.csv) (Chrome primary), [`data/latency-webkit.json`](data/latency-webkit.json), > [`data/combined.json`](data/combined.json). --- ## 1. Environment **Hardware / OS.** Apple M4 Max, 16 logical cores, 48 GB RAM, macOS (darwin 25.5.0), arm64. **Browser runtime (the runtime that decides the swap).** System Google Chrome driven headed/headless via Playwright 1.57.0; onnxruntime-web **1.24.3** (the repo's pinned version, `node_modules/onnxruntime-web`); WASM CPU execution provider; SIMD on. Also spot-checked on WebKit (Safari engine) via Playwright. **Conversion toolchain (Python, throwaway venv — NOT an app dependency).** Python 3.12, `onnx==1.17.0`, `onnxruntime==1.22.0` (Python, used only for conversion + the numerical-equivalence reference), `onnxconverter-common==1.14.0`, `onnxslim==0.1.95`, `opencv-python-headless==5.0.0.93`, `numpy==2.2.6`. Full pin list: [`requirements.txt`](../../scripts/model-optimization/requirements.txt). **Why Python latency is not used as evidence.** In onnxruntime-web the `cpu` EP *is* the WebAssembly backend, with different kernels and threading than native Python onnxruntime. Every latency number in this report comes from onnxruntime-web in a real browser. --- ## 2. Method 1. **Baseline.** Download the two pinned FP32 checkpoints, verify SHA-256, characterize each (opset, node/initializer counts, weight dtypes, external-data, I/O contract, op histogram). 2. **Variants.** Build a matrix per model, each **preserving the exact public I/O contract** (see [§3](#3-hard-integration-constraints)). 3. **Numerical equivalence.** Feed byte-identical inputs to FP32 baseline and each candidate via Python onnxruntime (CPU) and measure raw-output error + end-to-end instance/box/mask agreement — with special attention to the decision boundaries (score ≈ 0.4, mask logit ≈ 0.0). 4. **Target-runtime latency.** Measure each variant in onnxruntime-web/WASM (session-create, cold, warm p50/p90/p95), configured exactly like the app's ECSeg session. **First** confirm each variant loads and runs at all (ONNX-checker passing is not sufficient). 5. **Quality.** Agreement-vs-FP32 on a held-out image set as the decision proxy (FP32 is the reference for a replacement decision); COCO calibration subset used for static INT8 and for diversity. See [§7 limitations](#7-limitations). 6. **Decide** per model (ADOPT / OPTIONAL / REJECT / INCONCLUSIVE). **Images.** Static-INT8 calibration: 50-image COCO val2017 subset (fixed id list, app preprocessing). Numerical-equivalence evaluation: the repo's 47 `datasets/` images — **disjoint** from calibration. --- ## 3. Hard integration constraints (read from the app source) These shape which optimizations are even admissible: - **Storage is uncompressed.** The offline model store writes the raw ONNX bytes into IndexedDB (`src/offline/services/offline-model-store.ts`). gzip/Brotli on the wire does **not** shrink the stored artifact — the download size must equal the pinned `sizeBytes`, and IndexedDB then holds the decompressed ONNX. **On-disk `.onnx` size is the real storage cost**, and is what the size tables below use. - **The I/O contract is verified strictly at curated import** — `verifyTensorAgainstSpec()` in `src/pages/.../custom-model-curated-flow.ts` checks each tensor's **dtype and shape** against the pinned contract: `images` f32 `[1,3,640,640]` → `labels` i64 `[1,300]`, `boxes` f32 `[1,300,4]`, `scores` f32 `[1,300]`, `masks` f32 `[1,300,160,160]`. A variant that changed an output dtype/shape would be rejected at setup. **Every variant here preserves this contract** (FP16 keeps I/O float32 via boundary casts; INT8 keeps I/O float). If a winning artifact is adopted, its `sizeBytes` + `sha256` in `src/core/models/custom-checkpoints.ts` must be updated too. - **ECSeg is pinned to the CPU/WASM EP** (`custom-checkpoints.ts`, `executionProviders: ['cpu']`) because the WebGPU EP fails this graph on ort-web 1.24.3. The graph contains 12 `GridSample` ops (deformable mask sampling), the most likely cause. **WebGPU is out of scope** — the benchmark measures CPU/WASM only. - **Session options** the harness mirrors: `executionProviders:['cpu']`, `graphOptimizationLevel:'all'`, `logSeverityLevel:3`, `executionMode` = `sequential` (1 thread) / `parallel` (pool), `env.wasm.simd=true`. --- ## 4. Baseline characterization Both checkpoints share the **identical graph topology** — 2753 nodes, 415 initializers, the same operator histogram — differing only in weight width (channel counts). They are the same architecture at two capacities. | | ECSeg-S | ECSeg-M | |---|---|---| | SHA-256 | `fd30a61c…2522da` | `ded81475…820474` | | Size on disk | 41,889,348 B (39.95 MiB) | 80,820,640 B (77.08 MiB) | | opset / IR | 17 / 8 | 17 / 8 | | Nodes / initializers | 2753 / 415 | 2753 / 415 | | Weight dtype | 414×FLOAT + 1×INT64 | 414×FLOAT + 1×INT64 | | Weight bytes | ~40.99 MB (98% of file) | ~79.82 MB (99% of file) | | External data | none (self-contained) | none | | ONNX checker | pass | pass | **Operators (both).** `Conv`×45, `MatMul`×152, `Gemm`×5 (the quantizable compute), plus `LayerNormalization`×40, `Sigmoid`×59, `Softmax`×26, `Erf`×14 (GELU), **`GridSample`×12**, `TopK`×3, `GatherElements`×4, `Einsum`×1, and 954 `Constant` nodes (foldable). Because the file is ~98–99% FLOAT32 weights, halving weight width (FP16) roughly halves the file and INT8 roughly quarters the *weight* portion. --- ## 5. Variant matrix & sizes All variants pass the ONNX checker and preserve the exact I/O contract of [§3](#3-hard-integration-constraints). | Variant | What it does | ECSeg-S | | ECSeg-M | | |---|---|--:|--:|--:|--:| | | | MiB | % fp32 | MiB | % fp32 | | **fp32** | baseline | 39.95 | 100% | 77.08 | 100% | | **graphopt** | onnxslim: constant-fold, dead-node elimination, safe fusions (runtime-agnostic) | 40.72 | 102% | 77.85 | 101% | | **fp16** | float16 weights, I/O kept float32 (boundary casts); conservative op block-list | 20.57 | 51% | 39.13 | 51% | | **int8-dynamic** | weight-only dynamic INT8 (per-channel) on MatMul/Gemm | 16.57 | 41% | 32.75 | 42% | | **int8-static** | static QDQ INT8 (Conv+MatMul), COCO calibration | 12.87 | 32% | 22.40 | 29% | | **int8-static-selective** | static QDQ INT8 but mask-head nodes kept float | 13.98 | 35% | 24.41 | 32% | Note: **graphopt is slightly larger** — onnxslim folds `Constant` nodes into initializers, materializing a few tensors. Its only potential benefit is faster session-create, not smaller storage (the app already applies `graphOptimizationLevel:'all'` at load, so the runtime graph is equivalent either way). --- ## 6. Results ### 6.1 Loads-and-runs in onnxruntime-web/WASM (the gate) **Single-thread WASM (`wasm-st`): every variant loads and runs.** All six S variants and six M variants create a session and produce the four outputs. Cross-runtime check: the FP32 baseline returns the **same 17 instances** on the reference image as Python onnxruntime — the harness is faithful. **Multi-thread WASM (`wasm-mt`, the cross-origin-isolated browser config): the ECSeg graph hard-hangs at `session.create` — for every variant, and for FP32 too.** This was isolated carefully: - Reproduced across **every** threaded configuration tried: `numThreads` auto(0) / 4 / 2, `executionMode` both `parallel` and `sequential`, both headless and headed, both `chromium.launch` and `launchPersistentContext`. All hang at session creation; the hang is a hard, synchronous block of the renderer (an in-page 30 s timeout never fires), i.e. not merely slow. - **It is ECSeg-graph-specific, not a harness artifact.** The repo's existing [`scripts/benchmark-sam2.mjs`](../../scripts/benchmark-sam2.mjs) harness — same onnxruntime-web 1.24.3, same single-file ORT bundle, same COOP/COEP server, same machine — brings the threaded WASM pool up fine and completes `wasm-mt` for SAM tiny/small/large. The ECSeg graph is what the threaded build cannot instantiate. - This is the **same class of failure the app already documents** for ECSeg on the WebGPU EP (hence the hard `executionProviders:['cpu']` pin): the graph's deformable-attention machinery — 12 `GridSample` ops, `GatherElements`, `Einsum`, dynamic `TopK` — is not handled by these ORT-web backends. **Consequences for this study.** 1. The benchmark measures **single-thread WASM**, which is not a fallback but the config the app actually ships on **Desktop (Tauri) and iOS** (both pinned to one thread by `session-parameters.ts`), i.e. 2 of 3 platforms. It is also the cleaner per-op comparison. 2. For the pure browser (`app.annotateit.ai`, isolated → `numThreads:0`), ECSeg would request exactly the threaded pool that hangs here. Whether the app's bundler wiring avoids the hang in production is **out of scope and untested** — but optimization does **not** change it (every variant retains the GridSample machinery and hangs identically), so it is orthogonal to the swap decision. 3. Latency is input-independent for this architecture (static 640×640 input, fixed 300 queries, fixed 160×160 masks, no data-dependent loops), so a single reference input is representative for timing. ### 6.2 Browser latency (onnxruntime-web 1.24.3, CPU/WASM single-thread) Apple M4 Max, Chrome, `wasm-st` (single thread — the config Desktop+iOS ship, and the only one the ECSeg graph runs in; see [§6.1](#61-loads-and-runs-in-onnxruntime-webwasm-the-gate)). 15 warm repeats after 2 warm-ups, identical input. Latency is input-independent for this static graph. **ECSeg-S** (baseline warm p50 = 1299 ms) | variant | session-create ms | cold ms | **warm p50 ms** | p90 | p95 | Δ warm vs fp32 | instances | |---|--:|--:|--:|--:|--:|--:|--:| | fp32 | 529 | 1368 | **1299** | 1306 | — | — | 17 | | graphopt | 435 | 1367 | **1300** | 1308 | — | +0.1% | 17 | | fp16 | 703 | 1437 | **1366** | 1377 | — | **+5.2%** | 17 | | int8-dynamic | — | — | **does not load** | — | — | — | — | | int8-static | 691 | 1625 | **1523** | 1535 | — | **+17.2%** | **0** | | int8-static-selective | 683 | 1604 | **1493** | 1500 | — | **+14.9%** | **0** | **ECSeg-M** (baseline warm p50 = 2093 ms) | variant | session-create ms | cold ms | **warm p50 ms** | p90 | p95 | Δ warm vs fp32 | instances | |---|--:|--:|--:|--:|--:|--:|--:| | fp32 | 531 | 2140 | **2093** | 2103 | — | — | 18 | | graphopt | 434 | 2135 | **2089** | 2093 | — | −0.2% | 18 | | fp16 | 693 | 2238 | **2177** | 2185 | — | **+4.0%** | 18 | | int8-dynamic | — | — | **does not load** | — | — | — | — | | int8-static | 695 | 2482 | **2378** | 2388 | — | **+13.6%** | **0** | | int8-static-selective | 673 | 2453 | **2343** | 2349 | — | **+11.9%** | **0** | **Reading the table.** - **graphopt**: warm latency identical; only a ~90 ms cheaper session-create; **larger** on disk. No win. - **fp16**: **slower** in WASM (+4–5%). The WASM CPU EP has no native fp16 compute kernels, so ORT inserts cast-to-fp32 around every fp16 op — smaller weights, more work. S breaches the ≤5% warm budget. - **int8-dynamic**: **fails `session.create`** in ort-web 1.24.3 — `[ShapeInferenceError] Incompatible dimensions` (ONNX-checker-valid but runtime-rejected). Hard fail. - **int8-static / -selective**: **slower** (+12–17%: QuantizeLinear/DequantizeLinear overhead with no optimized INT8 kernels for this graph outweighs the smaller weights) **and produce 0 instances** — the head quantization collapses `scores` below the 0.4 threshold (calibration hit a zero-range weight in the segmentation head → zero scale → degenerate output; see [§6.3](#63-numerical-equivalence-vs-fp32)). Keeping the mask head float (`-selective`) does not recover detection — the score/box heads are also destroyed. Net: the two variants that reduce storage the most (INT8) are the worst on both latency and quality; the one with acceptable quality (fp16) is slower; the only lossless one (graphopt) is larger. **No variant is simultaneously smaller, no-slower, and quality-neutral.** ### 6.3 Numerical equivalence vs FP32 Candidate vs FP32 on identical inputs (Python onnxruntime CPU), 47 held-out `datasets/` images, 358 (S) / ~360 (M) FP32 instances total. `mean mask IoU` is over matched instances; `worst-img` and `min-inst` expose the tail. (`label_agreement_all300` is omitted — it is dominated by sub-threshold junk queries and is not meaningful; `class match on real detections` is 1.00 for every runnable variant.) **ECSeg-S** | variant | cand/base inst | mean mask IoU | worst-img mask IoU | min-inst mask IoU | mask pixel flips | near-0.4 flips | NaN/Inf | |---|--:|--:|--:|--:|--:|--:|:--:| | graphopt | 358/358 | **1.000** | 1.000 | 1.000 | 0.0% | 0 | no | | fp16 | 357/358 | 0.988 | 0.667 | 0.000 | 13.9% | 1 | no | | fp16-hi | 357/358 | 0.985 | 0.667 | 0.000 | 14.0% | 1 | no | | int8-dynamic | — | — (fails to load in ORT-web) | | | | | | | int8-static | **0**/358 | n/a | n/a | n/a | — | 358 | no | | int8-static-selective | **0**/358 | n/a | n/a | n/a | — | 358 | no | **ECSeg-M** (341 FP32 instances total) | variant | cand/base inst | mean mask IoU | worst-img mask IoU | min-inst mask IoU | mask pixel flips | near-0.4 flips | NaN/Inf | |---|--:|--:|--:|--:|--:|--:|:--:| | graphopt | 341/341 | **1.000** | 1.000 | 1.000 | 0.0% | 0 | no | | fp16 | 344/341 | 0.979 | 0.613 | 0.000 | 13.5% | 5 | no | | fp16-hi | 342/341 | 0.976 | 0.548 | 0.000 | 13.3% | 5 | no | | int8-dynamic | — | — (fails to load in ORT-web **and** Python ORT) | | | | | | | int8-static | **0**/341 | n/a | n/a | n/a | — | 341 | no | | int8-static-selective | **0**/341 | n/a | n/a | n/a | — | 341 | no | M fp16 quality is **worse** than S (mean 0.976–0.979 vs 0.985–0.988; worst-image 0.55–0.61 vs 0.67) — the larger model's larger mask logits (abs error up to ~670) degrade more under fp16. **Reading it.** - **graphopt is bit-identical** to FP32 (as expected from a lossless graph rewrite). - **fp16 / fp16-hi**: mean mask IoU ~0.985–0.988 looks high, but the tail is real — at least one instance drops to IoU 0.0 (a fully wrong mask) and the worst image averages 0.67. Raising the fp16 range clamp from the library default ±1e4 to fp16's true ±65504 (`fp16-hi`) does **not** help — the error is inherent fp16 precision loss in the large mask logits, not clipping (max mask-logit error stays ~300). For an annotation tool this occasional fully-wrong mask is a real UX cost. - **int8-static / -selective**: **0 instances** — all detections drop below the 0.4 score threshold (358 near-threshold decision flips; no NaN/Inf — the head simply collapses). Static INT8 calibration hit a zero-range weight in the segmentation head (zero scale → degenerate quantized output). Excluding the mask head (`-selective`) does not recover it — the score/box heads are equally affected. - **int8-dynamic**: does not load in **either** runtime — ort-web 1.24.3 rejects it with `[ShapeInferenceError] Incompatible dimensions` and Python onnxruntime 1.22 fails to load the model at all. Per-channel dynamic quantization of this graph's MatMuls produces a structurally invalid graph (ONNX-checker-valid, yet unloadable) — a concrete instance of "the checker accepting it is not sufficient". ### 6.3.1 Side-by-side visualizations Rendered by `scripts/model-optimization/visualize.py` (FP32 vs candidate, plus a union-mask disagreement panel). Stored under [`img/`](img/): - [`ecseg-s-int8static-empty.png`](img/ecseg-s-int8static-empty.png) — the failure that matters most: FP32 finds 18 instances on a parking-lot image, **int8-static finds 0** (union mask IoU 0.000). This is what "smaller" costs. - [`ecseg-s-fp16-worstcase.png`](img/ecseg-s-fp16-worstcase.png) — FP32 vs fp16 on the worst held-out image for fp16 (per-instance mean mask IoU 0.667). Overall coverage is close (union IoU high), but individual instance boundaries drift — the per-instance metric is where fp16 loses. - [`ecseg-m-fp16-typical.png`](img/ecseg-m-fp16-typical.png) — a typical image where M fp16 matches FP32 (union IoU 1.0). fp16's damage is concentrated on a minority of images, not uniform. ### 6.4 Cross-runtime & cross-browser correctness The FP32 baseline and the lossless graphopt return the **same instances** in ORT-web/WASM as in Python onnxruntime (17 on the S reference image, 18 on M), confirming the harness is faithful and the WASM CPU EP matches the reference. The broken INT8 variants return **0 instances in both** runtimes — a consistent (if useless) result, not a runtime discrepancy. **Cross-browser (WebKit / Safari engine, single-thread WASM, ECSeg-S).** Every Chrome finding reproduces on WebKit — the conclusions are not a Blink quirk: | variant | WebKit warm p50 | instances | vs Chrome | |---|--:|--:|---| | fp32 | 1360 ms | 17 | same behavior (Chrome 1299 ms) | | fp16 | 1455 ms (**+7%** vs its own fp32) | 17 | slower, same as Chrome | | int8-dynamic | **does not load** (`session.create` fails) | — | same as Chrome | | int8-static | 1704 ms (**+25%**) | **0** | slower + empty, same as Chrome | --- ## 7. Limitations - **Quality proxy, not full COCO AP.** The decision metric is agreement-vs-FP32 (FP32 is the reference for a drop-in replacement) on a 47-image held-out set, plus a 50-image COCO calibration set. True mask/box AP against COCO ground truth (with pycocotools) is not computed here; the quality budget (≤0.5 AP drop) is therefore evaluated via strict proxies (mask IoU vs FP32, near-threshold decision flips), and any verdict resting on the proxy is marked accordingly. - **One machine, two browser engines.** All numbers are Apple M4 Max; Chrome (Blink) is primary and WebKit (Safari engine) confirms the four decisive S cases ([§6.4](#64-cross-runtime--cross-browser-correctness)). No Windows / Android / low-end-device numbers — but the verdict rests on *relative* deltas (smaller-but- slower, won't-load, 0-instances) that are architectural, not device-specific, so they are unlikely to invert on other hardware. - **Multi-thread WASM could not be measured** for the ECSeg graph (it hard-hangs `session.create`); see [§6.1](#61-loads-and-runs-in-onnxruntime-webwasm-the-gate). Single-thread is the config that runs, and is what Desktop + iOS ship. Whether the pure-browser multi-thread path works in production is a separate, pre-existing question independent of this optimization decision. - **Because no variant qualified, no adoption/rollback path was exercised.** The registry (`custom-checkpoints.ts`), IndexedDB save/load, curated-import verification, Test flow and real inference were left on FP32 (the ECSeg jest suites — 63 tests — still pass; repo type-check is green). --- ## 8. Verdicts ### ECSeg-S — **REJECT** No variant is simultaneously smaller, no-slower, and quality-neutral in the target runtime: | candidate | size | warm latency | quality | loads/runs | why not | |---|---|---|---|---|---| | graphopt | **+2% (larger)** | ≈ equal | bit-exact | yes | no size or latency win; only ~90 ms cheaper create | | fp16 / fp16-hi | 51% | **+5.2% slower** | mask IoU 0.985–0.988 mean but 0.0 min-inst / 0.67 worst-img, 14% flips | yes | slower **and** occasional fully-wrong masks; over the 5% budget | | int8-dynamic | 41% | — | — | **NO** (`ShapeInferenceError`) | does not create a session in ort-web 1.24.3 | | int8-static | 32% | **+17% slower** | **0 instances** | yes (but useless) | head collapses; slower too | | int8-static-selective | 35% | **+15% slower** | **0 instances** | yes (but useless) | head collapses; slower too | Keep the FP32 checkpoint. The only lossless option is larger; every smaller option is slower and/or broken. ### ECSeg-M — **REJECT** Same picture, and fp16's quality is **worse** on M than on S (larger logits degrade more under fp16): | candidate | size | warm latency | quality | loads/runs | why not | |---|---|---|---|---|---| | graphopt | +1% (larger) | ≈ equal | bit-exact | yes | no win | | fp16 / fp16-hi | 49% | **+4.0% slower** | mask IoU 0.976 mean but 0.0 min-inst / **0.55 worst-img**, 13% flips | yes | slower **and** worse masks than S; a segmentation-quality regression the product should not ship for a size-only gain | | int8-dynamic | 42% | — | — | **NO** (`ShapeInferenceError`) | does not create a session | | int8-static | 29% | **+14% slower** | **0 instances** | yes (but useless) | head collapses; slower too | | int8-static-selective | 32% | **+12% slower** | **0 instances** | yes (but useless) | head collapses; slower too | M fp16 sits just inside the 5% latency budget, so it is the single "closest" candidate — but it buys a 49% storage cut with a **latency regression and a measurable mask-quality regression** (worst-image mask IoU 0.55, occasional IoU-0 instances), and AnnotateIt weights annotation quality and latency above compression. Not worth adopting, even as a device-conditional OPTIONAL. Keep the FP32 checkpoint. ### Why every path failed — root cause The ECSeg head is a DETR-style deformable-attention segmentation head (12 `GridSample`, `Einsum`, dynamic `TopK`, `GatherElements`). Three independent facts fall out of that one design: 1. **It won't run on the threaded WASM pool or WebGPU** — only single-thread CPU/WASM. (Already why the app pins `['cpu']`.) 2. **Its mask logits have a wide dynamic range** (errors up to ~670 in M), so **fp16 loses real precision** exactly where it matters — near the mask-logit=0 decision boundary — flipping pixels. 3. **Its score/box heads have low-rank / near-constant weight tensors**, so **static-INT8 MinMax calibration produces zero-scale quantizers** that collapse the output to nothing. And because the graph is ~98% weights with almost no INT8-friendly large-matmul structure that ort-web has fast kernels for, the QDQ/cast overhead makes the quantized graphs **slower**, not faster, on WASM. Smaller file, more runtime work: the exact anti-pattern the study was asked to rule out. --- ## 9. Reproduce See [`scripts/model-optimization/README.md`](../../scripts/model-optimization/README.md) for the exact commands (download+verify, build variants, correctness, browser benchmark, aggregate).