| # ECSeg-S / ECSeg-M ONNX optimization & quantization report |
|
|
| **Question.** Should AnnotateIt replace the shipped FP32 ECSeg-S / ECSeg-M instance-segmentation |
| checkpoints with an optimized (graph-optimized / FP16 / INT8) variant β judged on size **and** speed |
| **and** quality **and** stability **and** browser compatibility together, not size alone? |
|
|
| **Decision: keep FP32 for both models (details in [Β§8](#8-verdicts)).** |
|
|
| | Model | Verdict | Best format | Size before β after | Browser warm latency before β after | Quality vs FP32 | |
| |---|---|---|---|---|---| |
| | **ECSeg-S** | **REJECT** | FP32 (unchanged) | 39.95 MiB β 39.95 MiB (no adoptable smaller variant) | 1299 ms β 1299 ms (every smaller variant is slower or won't run) | unchanged (candidates lose quality or produce 0 instances) | |
| | **ECSeg-M** | **REJECT** | FP32 (unchanged) | 77.08 MiB β 77.08 MiB | 2093 ms β 2093 ms | unchanged | |
|
|
| Nothing was changed in the app or in the checkpoint registry β there is no winning variant. The reusable |
| conversion + browser-benchmark tooling and this report are the application-repository deliverables. |
|
|
| > Reproduce everything: [`scripts/model-optimization/README.md`](../../scripts/model-optimization/README.md). |
| > Generated `.onnx` variants are **not** committed (large, rebuildable). |
| > Machine-readable results: [`data/sizes.csv`](data/sizes.csv), [`data/correctness.csv`](data/correctness.csv), |
| > [`data/latency.csv`](data/latency.csv) (Chrome primary), [`data/latency-webkit.json`](data/latency-webkit.json), |
| > [`data/combined.json`](data/combined.json). |
|
|
| --- |
|
|
| ## 1. Environment |
|
|
| **Hardware / OS.** Apple M4 Max, 16 logical cores, 48 GB RAM, macOS (darwin 25.5.0), arm64. |
|
|
| **Browser runtime (the runtime that decides the swap).** System Google Chrome driven headed/headless via |
| Playwright 1.57.0; onnxruntime-web **1.24.3** (the repo's pinned version, `node_modules/onnxruntime-web`); |
| WASM CPU execution provider; SIMD on. Also spot-checked on WebKit (Safari engine) via Playwright. |
|
|
| **Conversion toolchain (Python, throwaway venv β NOT an app dependency).** Python 3.12, |
| `onnx==1.17.0`, `onnxruntime==1.22.0` (Python, used only for conversion + the numerical-equivalence |
| reference), `onnxconverter-common==1.14.0`, `onnxslim==0.1.95`, `opencv-python-headless==5.0.0.93`, |
| `numpy==2.2.6`. Full pin list: [`requirements.txt`](../../scripts/model-optimization/requirements.txt). |
|
|
| **Why Python latency is not used as evidence.** In onnxruntime-web the `cpu` EP *is* the WebAssembly |
| backend, with different kernels and threading than native Python onnxruntime. Every latency number in this |
| report comes from onnxruntime-web in a real browser. |
|
|
| --- |
|
|
| ## 2. Method |
|
|
| 1. **Baseline.** Download the two pinned FP32 checkpoints, verify SHA-256, characterize each |
| (opset, node/initializer counts, weight dtypes, external-data, I/O contract, op histogram). |
| 2. **Variants.** Build a matrix per model, each **preserving the exact public I/O contract** |
| (see [Β§3](#3-hard-integration-constraints)). |
| 3. **Numerical equivalence.** Feed byte-identical inputs to FP32 baseline and each candidate via Python |
| onnxruntime (CPU) and measure raw-output error + end-to-end instance/box/mask agreement β with special |
| attention to the decision boundaries (score β 0.4, mask logit β 0.0). |
| 4. **Target-runtime latency.** Measure each variant in onnxruntime-web/WASM (session-create, cold, warm |
| p50/p90/p95), configured exactly like the app's ECSeg session. **First** confirm each variant loads and |
| runs at all (ONNX-checker passing is not sufficient). |
| 5. **Quality.** Agreement-vs-FP32 on a held-out image set as the decision proxy (FP32 is the reference for |
| a replacement decision); COCO calibration subset used for static INT8 and for diversity. See |
| [Β§7 limitations](#7-limitations). |
| 6. **Decide** per model (ADOPT / OPTIONAL / REJECT / INCONCLUSIVE). |
|
|
| **Images.** Static-INT8 calibration: 50-image COCO val2017 subset (fixed id list, app preprocessing). |
| Numerical-equivalence evaluation: the repo's 47 `datasets/` images β **disjoint** from calibration. |
|
|
| --- |
|
|
| ## 3. Hard integration constraints (read from the app source) |
|
|
| These shape which optimizations are even admissible: |
|
|
| - **Storage is uncompressed.** The offline model store writes the raw ONNX bytes into IndexedDB |
| (`src/offline/services/offline-model-store.ts`). gzip/Brotli on the wire does **not** shrink the stored |
| artifact β the download size must equal the pinned `sizeBytes`, and IndexedDB then holds the decompressed |
| ONNX. **On-disk `.onnx` size is the real storage cost**, and is what the size tables below use. |
| - **The I/O contract is verified strictly at curated import** β `verifyTensorAgainstSpec()` in |
| `src/pages/.../custom-model-curated-flow.ts` checks each tensor's **dtype and shape** against the pinned |
| contract: `images` f32 `[1,3,640,640]` β `labels` i64 `[1,300]`, `boxes` f32 `[1,300,4]`, `scores` f32 |
| `[1,300]`, `masks` f32 `[1,300,160,160]`. A variant that changed an output dtype/shape would be rejected |
| at setup. **Every variant here preserves this contract** (FP16 keeps I/O float32 via boundary casts; INT8 |
| keeps I/O float). If a winning artifact is adopted, its `sizeBytes` + `sha256` in |
| `src/core/models/custom-checkpoints.ts` must be updated too. |
| - **ECSeg is pinned to the CPU/WASM EP** (`custom-checkpoints.ts`, `executionProviders: ['cpu']`) because |
| the WebGPU EP fails this graph on ort-web 1.24.3. The graph contains 12 `GridSample` ops (deformable mask |
| sampling), the most likely cause. **WebGPU is out of scope** β the benchmark measures CPU/WASM only. |
| - **Session options** the harness mirrors: `executionProviders:['cpu']`, `graphOptimizationLevel:'all'`, |
| `logSeverityLevel:3`, `executionMode` = `sequential` (1 thread) / `parallel` (pool), `env.wasm.simd=true`. |
|
|
| --- |
|
|
| ## 4. Baseline characterization |
|
|
| Both checkpoints share the **identical graph topology** β 2753 nodes, 415 initializers, the same operator |
| histogram β differing only in weight width (channel counts). They are the same architecture at two |
| capacities. |
|
|
| | | ECSeg-S | ECSeg-M | |
| |---|---|---| |
| | SHA-256 | `fd30a61cβ¦2522da` | `ded81475β¦820474` | |
| | Size on disk | 41,889,348 B (39.95 MiB) | 80,820,640 B (77.08 MiB) | |
| | opset / IR | 17 / 8 | 17 / 8 | |
| | Nodes / initializers | 2753 / 415 | 2753 / 415 | |
| | Weight dtype | 414ΓFLOAT + 1ΓINT64 | 414ΓFLOAT + 1ΓINT64 | |
| | Weight bytes | ~40.99 MB (98% of file) | ~79.82 MB (99% of file) | |
| | External data | none (self-contained) | none | |
| | ONNX checker | pass | pass | |
|
|
| **Operators (both).** `Conv`Γ45, `MatMul`Γ152, `Gemm`Γ5 (the quantizable compute), plus |
| `LayerNormalization`Γ40, `Sigmoid`Γ59, `Softmax`Γ26, `Erf`Γ14 (GELU), **`GridSample`Γ12**, `TopK`Γ3, |
| `GatherElements`Γ4, `Einsum`Γ1, and 954 `Constant` nodes (foldable). Because the file is ~98β99% FLOAT32 |
| weights, halving weight width (FP16) roughly halves the file and INT8 roughly quarters the *weight* portion. |
|
|
| --- |
|
|
| ## 5. Variant matrix & sizes |
|
|
| All variants pass the ONNX checker and preserve the exact I/O contract of [Β§3](#3-hard-integration-constraints). |
|
|
| | Variant | What it does | ECSeg-S | | ECSeg-M | | |
| |---|---|--:|--:|--:|--:| |
| | | | MiB | % fp32 | MiB | % fp32 | |
| | **fp32** | baseline | 39.95 | 100% | 77.08 | 100% | |
| | **graphopt** | onnxslim: constant-fold, dead-node elimination, safe fusions (runtime-agnostic) | 40.72 | 102% | 77.85 | 101% | |
| | **fp16** | float16 weights, I/O kept float32 (boundary casts); conservative op block-list | 20.57 | 51% | 39.13 | 51% | |
| | **int8-dynamic** | weight-only dynamic INT8 (per-channel) on MatMul/Gemm | 16.57 | 41% | 32.75 | 42% | |
| | **int8-static** | static QDQ INT8 (Conv+MatMul), COCO calibration | 12.87 | 32% | 22.40 | 29% | |
| | **int8-static-selective** | static QDQ INT8 but mask-head nodes kept float | 13.98 | 35% | 24.41 | 32% | |
|
|
| Note: **graphopt is slightly larger** β onnxslim folds `Constant` nodes into initializers, materializing a |
| few tensors. Its only potential benefit is faster session-create, not smaller storage (the app already |
| applies `graphOptimizationLevel:'all'` at load, so the runtime graph is equivalent either way). |
|
|
| --- |
|
|
| ## 6. Results |
|
|
| ### 6.1 Loads-and-runs in onnxruntime-web/WASM (the gate) |
|
|
| **Single-thread WASM (`wasm-st`): every variant loads and runs.** All six S variants and six M variants |
| create a session and produce the four outputs. Cross-runtime check: the FP32 baseline returns the **same 17 |
| instances** on the reference image as Python onnxruntime β the harness is faithful. |
|
|
| **Multi-thread WASM (`wasm-mt`, the cross-origin-isolated browser config): the ECSeg graph hard-hangs at |
| `session.create` β for every variant, and for FP32 too.** This was isolated carefully: |
|
|
| - Reproduced across **every** threaded configuration tried: `numThreads` auto(0) / 4 / 2, `executionMode` |
| both `parallel` and `sequential`, both headless and headed, both `chromium.launch` and |
| `launchPersistentContext`. All hang at session creation; the hang is a hard, synchronous block of the |
| renderer (an in-page 30 s timeout never fires), i.e. not merely slow. |
| - **It is ECSeg-graph-specific, not a harness artifact.** The repo's existing |
| [`scripts/benchmark-sam2.mjs`](../../scripts/benchmark-sam2.mjs) harness β same onnxruntime-web 1.24.3, |
| same single-file ORT bundle, same COOP/COEP server, same machine β brings the threaded WASM pool up |
| fine and completes `wasm-mt` for SAM tiny/small/large. The ECSeg graph is what the threaded build cannot |
| instantiate. |
| - This is the **same class of failure the app already documents** for ECSeg on the WebGPU EP (hence the |
| hard `executionProviders:['cpu']` pin): the graph's deformable-attention machinery β 12 `GridSample` |
| ops, `GatherElements`, `Einsum`, dynamic `TopK` β is not handled by these ORT-web backends. |
|
|
| **Consequences for this study.** |
| 1. The benchmark measures **single-thread WASM**, which is not a fallback but the config the app actually |
| ships on **Desktop (Tauri) and iOS** (both pinned to one thread by `session-parameters.ts`), i.e. 2 of |
| 3 platforms. It is also the cleaner per-op comparison. |
| 2. For the pure browser (`app.annotateit.ai`, isolated β `numThreads:0`), ECSeg would request exactly the |
| threaded pool that hangs here. Whether the app's bundler wiring avoids the hang in production is **out of |
| scope and untested** β but optimization does **not** change it (every variant retains the GridSample |
| machinery and hangs identically), so it is orthogonal to the swap decision. |
| 3. Latency is input-independent for this architecture (static 640Γ640 input, fixed 300 queries, fixed |
| 160Γ160 masks, no data-dependent loops), so a single reference input is representative for timing. |
|
|
| ### 6.2 Browser latency (onnxruntime-web 1.24.3, CPU/WASM single-thread) |
|
|
| Apple M4 Max, Chrome, `wasm-st` (single thread β the config Desktop+iOS ship, and the only one the ECSeg |
| graph runs in; see [Β§6.1](#61-loads-and-runs-in-onnxruntime-webwasm-the-gate)). 15 warm repeats after 2 |
| warm-ups, identical input. Latency is input-independent for this static graph. |
|
|
| **ECSeg-S** (baseline warm p50 = 1299 ms) |
|
|
| | variant | session-create ms | cold ms | **warm p50 ms** | p90 | p95 | Ξ warm vs fp32 | instances | |
| |---|--:|--:|--:|--:|--:|--:|--:| |
| | fp32 | 529 | 1368 | **1299** | 1306 | β | β | 17 | |
| | graphopt | 435 | 1367 | **1300** | 1308 | β | +0.1% | 17 | |
| | fp16 | 703 | 1437 | **1366** | 1377 | β | **+5.2%** | 17 | |
| | int8-dynamic | β | β | **does not load** | β | β | β | β | |
| | int8-static | 691 | 1625 | **1523** | 1535 | β | **+17.2%** | **0** | |
| | int8-static-selective | 683 | 1604 | **1493** | 1500 | β | **+14.9%** | **0** | |
|
|
| **ECSeg-M** (baseline warm p50 = 2093 ms) |
|
|
| | variant | session-create ms | cold ms | **warm p50 ms** | p90 | p95 | Ξ warm vs fp32 | instances | |
| |---|--:|--:|--:|--:|--:|--:|--:| |
| | fp32 | 531 | 2140 | **2093** | 2103 | β | β | 18 | |
| | graphopt | 434 | 2135 | **2089** | 2093 | β | β0.2% | 18 | |
| | fp16 | 693 | 2238 | **2177** | 2185 | β | **+4.0%** | 18 | |
| | int8-dynamic | β | β | **does not load** | β | β | β | β | |
| | int8-static | 695 | 2482 | **2378** | 2388 | β | **+13.6%** | **0** | |
| | int8-static-selective | 673 | 2453 | **2343** | 2349 | β | **+11.9%** | **0** | |
|
|
| **Reading the table.** |
| - **graphopt**: warm latency identical; only a ~90 ms cheaper session-create; **larger** on disk. No win. |
| - **fp16**: **slower** in WASM (+4β5%). The WASM CPU EP has no native fp16 compute kernels, so ORT inserts |
| cast-to-fp32 around every fp16 op β smaller weights, more work. S breaches the β€5% warm budget. |
| - **int8-dynamic**: **fails `session.create`** in ort-web 1.24.3 β |
| `[ShapeInferenceError] Incompatible dimensions` (ONNX-checker-valid but runtime-rejected). Hard fail. |
| - **int8-static / -selective**: **slower** (+12β17%: QuantizeLinear/DequantizeLinear overhead with no |
| optimized INT8 kernels for this graph outweighs the smaller weights) **and produce 0 instances** β the |
| head quantization collapses `scores` below the 0.4 threshold (calibration hit a zero-range weight in the |
| segmentation head β zero scale β degenerate output; see [Β§6.3](#63-numerical-equivalence-vs-fp32)). |
| Keeping the mask head float (`-selective`) does not recover detection β the score/box heads are also |
| destroyed. |
|
|
| Net: the two variants that reduce storage the most (INT8) are the worst on both latency and quality; the |
| one with acceptable quality (fp16) is slower; the only lossless one (graphopt) is larger. **No variant is |
| simultaneously smaller, no-slower, and quality-neutral.** |
|
|
| ### 6.3 Numerical equivalence vs FP32 |
|
|
| Candidate vs FP32 on identical inputs (Python onnxruntime CPU), 47 held-out `datasets/` images, 358 (S) / |
| ~360 (M) FP32 instances total. `mean mask IoU` is over matched instances; `worst-img` and `min-inst` expose |
| the tail. (`label_agreement_all300` is omitted β it is dominated by sub-threshold junk queries and is not |
| meaningful; `class match on real detections` is 1.00 for every runnable variant.) |
|
|
| **ECSeg-S** |
|
|
| | variant | cand/base inst | mean mask IoU | worst-img mask IoU | min-inst mask IoU | mask pixel flips | near-0.4 flips | NaN/Inf | |
| |---|--:|--:|--:|--:|--:|--:|:--:| |
| | graphopt | 358/358 | **1.000** | 1.000 | 1.000 | 0.0% | 0 | no | |
| | fp16 | 357/358 | 0.988 | 0.667 | 0.000 | 13.9% | 1 | no | |
| | fp16-hi | 357/358 | 0.985 | 0.667 | 0.000 | 14.0% | 1 | no | |
| | int8-dynamic | β | β (fails to load in ORT-web) | | | | | | |
| | int8-static | **0**/358 | n/a | n/a | n/a | β | 358 | no | |
| | int8-static-selective | **0**/358 | n/a | n/a | n/a | β | 358 | no | |
|
|
| **ECSeg-M** (341 FP32 instances total) |
|
|
| | variant | cand/base inst | mean mask IoU | worst-img mask IoU | min-inst mask IoU | mask pixel flips | near-0.4 flips | NaN/Inf | |
| |---|--:|--:|--:|--:|--:|--:|:--:| |
| | graphopt | 341/341 | **1.000** | 1.000 | 1.000 | 0.0% | 0 | no | |
| | fp16 | 344/341 | 0.979 | 0.613 | 0.000 | 13.5% | 5 | no | |
| | fp16-hi | 342/341 | 0.976 | 0.548 | 0.000 | 13.3% | 5 | no | |
| | int8-dynamic | β | β (fails to load in ORT-web **and** Python ORT) | | | | | | |
| | int8-static | **0**/341 | n/a | n/a | n/a | β | 341 | no | |
| | int8-static-selective | **0**/341 | n/a | n/a | n/a | β | 341 | no | |
|
|
| M fp16 quality is **worse** than S (mean 0.976β0.979 vs 0.985β0.988; worst-image 0.55β0.61 vs 0.67) β the |
| larger model's larger mask logits (abs error up to ~670) degrade more under fp16. |
|
|
| **Reading it.** |
| - **graphopt is bit-identical** to FP32 (as expected from a lossless graph rewrite). |
| - **fp16 / fp16-hi**: mean mask IoU ~0.985β0.988 looks high, but the tail is real β at least one instance |
| drops to IoU 0.0 (a fully wrong mask) and the worst image averages 0.67. Raising the fp16 range clamp |
| from the library default Β±1e4 to fp16's true Β±65504 (`fp16-hi`) does **not** help β the error is inherent |
| fp16 precision loss in the large mask logits, not clipping (max mask-logit error stays ~300). For an |
| annotation tool this occasional fully-wrong mask is a real UX cost. |
| - **int8-static / -selective**: **0 instances** β all detections drop below the 0.4 score threshold (358 |
| near-threshold decision flips; no NaN/Inf β the head simply collapses). Static INT8 calibration hit a |
| zero-range weight in the segmentation head (zero scale β degenerate quantized output). Excluding the mask |
| head (`-selective`) does not recover it β the score/box heads are equally affected. |
| - **int8-dynamic**: does not load in **either** runtime β ort-web 1.24.3 rejects it with |
| `[ShapeInferenceError] Incompatible dimensions` and Python onnxruntime 1.22 fails to load the model at |
| all. Per-channel dynamic quantization of this graph's MatMuls produces a structurally invalid graph |
| (ONNX-checker-valid, yet unloadable) β a concrete instance of "the checker accepting it is not |
| sufficient". |
|
|
| ### 6.3.1 Side-by-side visualizations |
|
|
| Rendered by `scripts/model-optimization/visualize.py` (FP32 vs candidate, plus a union-mask disagreement |
| panel). Stored under [`img/`](img/): |
|
|
| - [`ecseg-s-int8static-empty.png`](img/ecseg-s-int8static-empty.png) β the failure that matters most: FP32 |
| finds 18 instances on a parking-lot image, **int8-static finds 0** (union mask IoU 0.000). This is what |
| "smaller" costs. |
| - [`ecseg-s-fp16-worstcase.png`](img/ecseg-s-fp16-worstcase.png) β FP32 vs fp16 on the worst held-out image |
| for fp16 (per-instance mean mask IoU 0.667). Overall coverage is close (union IoU high), but individual |
| instance boundaries drift β the per-instance metric is where fp16 loses. |
| - [`ecseg-m-fp16-typical.png`](img/ecseg-m-fp16-typical.png) β a typical image where M fp16 matches FP32 |
| (union IoU 1.0). fp16's damage is concentrated on a minority of images, not uniform. |
|
|
| ### 6.4 Cross-runtime & cross-browser correctness |
|
|
| The FP32 baseline and the lossless graphopt return the **same instances** in ORT-web/WASM as in Python |
| onnxruntime (17 on the S reference image, 18 on M), confirming the harness is faithful and the WASM CPU EP |
| matches the reference. The broken INT8 variants return **0 instances in both** runtimes β a consistent |
| (if useless) result, not a runtime discrepancy. |
|
|
| **Cross-browser (WebKit / Safari engine, single-thread WASM, ECSeg-S).** Every Chrome finding reproduces on |
| WebKit β the conclusions are not a Blink quirk: |
|
|
| | variant | WebKit warm p50 | instances | vs Chrome | |
| |---|--:|--:|---| |
| | fp32 | 1360 ms | 17 | same behavior (Chrome 1299 ms) | |
| | fp16 | 1455 ms (**+7%** vs its own fp32) | 17 | slower, same as Chrome | |
| | int8-dynamic | **does not load** (`session.create` fails) | β | same as Chrome | |
| | int8-static | 1704 ms (**+25%**) | **0** | slower + empty, same as Chrome | |
|
|
| --- |
|
|
| ## 7. Limitations |
|
|
| - **Quality proxy, not full COCO AP.** The decision metric is agreement-vs-FP32 (FP32 is the reference for a |
| drop-in replacement) on a 47-image held-out set, plus a 50-image COCO calibration set. True mask/box AP |
| against COCO ground truth (with pycocotools) is not computed here; the quality budget (β€0.5 AP drop) is |
| therefore evaluated via strict proxies (mask IoU vs FP32, near-threshold decision flips), and any verdict |
| resting on the proxy is marked accordingly. |
| - **One machine, two browser engines.** All numbers are Apple M4 Max; Chrome (Blink) is primary and WebKit |
| (Safari engine) confirms the four decisive S cases ([Β§6.4](#64-cross-runtime--cross-browser-correctness)). |
| No Windows / Android / low-end-device numbers β but the verdict rests on *relative* deltas (smaller-but- |
| slower, won't-load, 0-instances) that are architectural, not device-specific, so they are unlikely to |
| invert on other hardware. |
| - **Multi-thread WASM could not be measured** for the ECSeg graph (it hard-hangs `session.create`); see |
| [Β§6.1](#61-loads-and-runs-in-onnxruntime-webwasm-the-gate). Single-thread is the config that runs, and is |
| what Desktop + iOS ship. Whether the pure-browser multi-thread path works in production is a separate, |
| pre-existing question independent of this optimization decision. |
| - **Because no variant qualified, no adoption/rollback path was exercised.** The registry |
| (`custom-checkpoints.ts`), IndexedDB save/load, curated-import verification, Test flow and real inference |
| were left on FP32 (the ECSeg jest suites β 63 tests β still pass; repo type-check is green). |
|
|
| --- |
|
|
| ## 8. Verdicts |
|
|
| ### ECSeg-S β **REJECT** |
|
|
| No variant is simultaneously smaller, no-slower, and quality-neutral in the target runtime: |
|
|
| | candidate | size | warm latency | quality | loads/runs | why not | |
| |---|---|---|---|---|---| |
| | graphopt | **+2% (larger)** | β equal | bit-exact | yes | no size or latency win; only ~90 ms cheaper create | |
| | fp16 / fp16-hi | 51% | **+5.2% slower** | mask IoU 0.985β0.988 mean but 0.0 min-inst / 0.67 worst-img, 14% flips | yes | slower **and** occasional fully-wrong masks; over the 5% budget | |
| | int8-dynamic | 41% | β | β | **NO** (`ShapeInferenceError`) | does not create a session in ort-web 1.24.3 | |
| | int8-static | 32% | **+17% slower** | **0 instances** | yes (but useless) | head collapses; slower too | |
| | int8-static-selective | 35% | **+15% slower** | **0 instances** | yes (but useless) | head collapses; slower too | |
|
|
| Keep the FP32 checkpoint. The only lossless option is larger; every smaller option is slower and/or |
| broken. |
|
|
| ### ECSeg-M β **REJECT** |
|
|
| Same picture, and fp16's quality is **worse** on M than on S (larger logits degrade more under fp16): |
|
|
| | candidate | size | warm latency | quality | loads/runs | why not | |
| |---|---|---|---|---|---| |
| | graphopt | +1% (larger) | β equal | bit-exact | yes | no win | |
| | fp16 / fp16-hi | 49% | **+4.0% slower** | mask IoU 0.976 mean but 0.0 min-inst / **0.55 worst-img**, 13% flips | yes | slower **and** worse masks than S; a segmentation-quality regression the product should not ship for a size-only gain | |
| | int8-dynamic | 42% | β | β | **NO** (`ShapeInferenceError`) | does not create a session | |
| | int8-static | 29% | **+14% slower** | **0 instances** | yes (but useless) | head collapses; slower too | |
| | int8-static-selective | 32% | **+12% slower** | **0 instances** | yes (but useless) | head collapses; slower too | |
|
|
| M fp16 sits just inside the 5% latency budget, so it is the single "closest" candidate β but it buys a 49% |
| storage cut with a **latency regression and a measurable mask-quality regression** (worst-image mask IoU |
| 0.55, occasional IoU-0 instances), and AnnotateIt weights annotation quality and latency above compression. |
| Not worth adopting, even as a device-conditional OPTIONAL. Keep the FP32 checkpoint. |
|
|
| ### Why every path failed β root cause |
|
|
| The ECSeg head is a DETR-style deformable-attention segmentation head (12 `GridSample`, `Einsum`, dynamic |
| `TopK`, `GatherElements`). Three independent facts fall out of that one design: |
|
|
| 1. **It won't run on the threaded WASM pool or WebGPU** β only single-thread CPU/WASM. (Already why the app |
| pins `['cpu']`.) |
| 2. **Its mask logits have a wide dynamic range** (errors up to ~670 in M), so **fp16 loses real precision** |
| exactly where it matters β near the mask-logit=0 decision boundary β flipping pixels. |
| 3. **Its score/box heads have low-rank / near-constant weight tensors**, so **static-INT8 MinMax |
| calibration produces zero-scale quantizers** that collapse the output to nothing. |
|
|
| And because the graph is ~98% weights with almost no INT8-friendly large-matmul structure that ort-web has |
| fast kernels for, the QDQ/cast overhead makes the quantized graphs **slower**, not faster, on WASM. Smaller |
| file, more runtime work: the exact anti-pattern the study was asked to rule out. |
|
|
| --- |
|
|
| ## 9. Reproduce |
|
|
| See [`scripts/model-optimization/README.md`](../../scripts/model-optimization/README.md) for the exact |
| commands (download+verify, build variants, correctness, browser benchmark, aggregate). |
|
|