report/ecseg-quantization-report.md

ECSeg-S / ECSeg-M ONNX optimization & quantization report

Question. Should AnnotateIt replace the shipped FP32 ECSeg-S / ECSeg-M instance-segmentation checkpoints with an optimized (graph-optimized / FP16 / INT8) variant — judged on size and speed and quality and stability and browser compatibility together, not size alone?

Decision: keep FP32 for both models (details in §8).

Model	Verdict	Best format	Size before → after	Browser warm latency before → after	Quality vs FP32
ECSeg-S	REJECT	FP32 (unchanged)	39.95 MiB → 39.95 MiB (no adoptable smaller variant)	1299 ms → 1299 ms (every smaller variant is slower or won't run)	unchanged (candidates lose quality or produce 0 instances)
ECSeg-M	REJECT	FP32 (unchanged)	77.08 MiB → 77.08 MiB	2093 ms → 2093 ms	unchanged

Nothing was changed in the app or in the checkpoint registry — there is no winning variant. The reusable conversion + browser-benchmark tooling and this report are the application-repository deliverables.

Reproduce everything: scripts/model-optimization/README.md. Generated .onnx variants are not committed (large, rebuildable). Machine-readable results: data/sizes.csv, data/correctness.csv, data/latency.csv (Chrome primary), data/latency-webkit.json, data/combined.json.

---

1. Environment

Hardware / OS. Apple M4 Max, 16 logical cores, 48 GB RAM, macOS (darwin 25.5.0), arm64.

Browser runtime (the runtime that decides the swap). System Google Chrome driven headed/headless via Playwright 1.57.0; onnxruntime-web 1.24.3 (the repo's pinned version, node_modules/onnxruntime-web); WASM CPU execution provider; SIMD on. Also spot-checked on WebKit (Safari engine) via Playwright.

Conversion toolchain (Python, throwaway venv — NOT an app dependency). Python 3.12, onnx==1.17.0, onnxruntime==1.22.0 (Python, used only for conversion + the numerical-equivalence reference), onnxconverter-common==1.14.0, onnxslim==0.1.95, opencv-python-headless==5.0.0.93, numpy==2.2.6. Full pin list: requirements.txt.

Why Python latency is not used as evidence. In onnxruntime-web the cpu EP is the WebAssembly backend, with different kernels and threading than native Python onnxruntime. Every latency number in this report comes from onnxruntime-web in a real browser.

---

2. Method

1. Baseline. Download the two pinned FP32 checkpoints, verify SHA-256, characterize each (opset, node/initializer counts, weight dtypes, external-data, I/O contract, op histogram).
2. Variants. Build a matrix per model, each preserving the exact public I/O contract (see §3).
3. Numerical equivalence. Feed byte-identical inputs to FP32 baseline and each candidate via Python onnxruntime (CPU) and measure raw-output error + end-to-end instance/box/mask agreement — with special attention to the decision boundaries (score ≈ 0.4, mask logit ≈ 0.0).
4. Target-runtime latency. Measure each variant in onnxruntime-web/WASM (session-create, cold, warm p50/p90/p95), configured exactly like the app's ECSeg session. First confirm each variant loads and runs at all (ONNX-checker passing is not sufficient).
5. Quality. Agreement-vs-FP32 on a held-out image set as the decision proxy (FP32 is the reference for a replacement decision); COCO calibration subset used for static INT8 and for diversity. See §7 limitations.
6. Decide per model (ADOPT / OPTIONAL / REJECT / INCONCLUSIVE).

Images. Static-INT8 calibration: 50-image COCO val2017 subset (fixed id list, app preprocessing). Numerical-equivalence evaluation: the repo's 47 datasets/ images — disjoint from calibration.

---

3. Hard integration constraints (read from the app source)

These shape which optimizations are even admissible:

• Storage is uncompressed. The offline model store writes the raw ONNX bytes into IndexedDB (src/offline/services/offline-model-store.ts). gzip/Brotli on the wire does not shrink the stored artifact — the download size must equal the pinned sizeBytes, and IndexedDB then holds the decompressed ONNX. On-disk .onnx size is the real storage cost, and is what the size tables below use.
• The I/O contract is verified strictly at curated import — verifyTensorAgainstSpec() in src/pages/.../custom-model-curated-flow.ts checks each tensor's dtype and shape against the pinned contract: images f32 [1,3,640,640] → labels i64 [1,300], boxes f32 [1,300,4], scores f32 [1,300], masks f32 [1,300,160,160]. A variant that changed an output dtype/shape would be rejected at setup. Every variant here preserves this contract (FP16 keeps I/O float32 via boundary casts; INT8 keeps I/O float). If a winning artifact is adopted, its sizeBytes + sha256 in src/core/models/custom-checkpoints.ts must be updated too.
• ECSeg is pinned to the CPU/WASM EP (custom-checkpoints.ts, executionProviders: ['cpu']) because the WebGPU EP fails this graph on ort-web 1.24.3. The graph contains 12 GridSample ops (deformable mask sampling), the most likely cause. WebGPU is out of scope — the benchmark measures CPU/WASM only.
• Session options the harness mirrors: executionProviders:['cpu'], graphOptimizationLevel:'all', logSeverityLevel:3, executionMode = sequential (1 thread) / parallel (pool), env.wasm.simd=true.

---

4. Baseline characterization

Both checkpoints share the identical graph topology — 2753 nodes, 415 initializers, the same operator histogram — differing only in weight width (channel counts). They are the same architecture at two capacities.

	ECSeg-S	ECSeg-M
SHA-256	fd30a61c…2522da	ded81475…820474
Size on disk	41,889,348 B (39.95 MiB)	80,820,640 B (77.08 MiB)
opset / IR	17 / 8	17 / 8
Nodes / initializers	2753 / 415	2753 / 415
Weight dtype	414×FLOAT + 1×INT64	414×FLOAT + 1×INT64
Weight bytes	~40.99 MB (98% of file)	~79.82 MB (99% of file)
External data	none (self-contained)	none
ONNX checker	pass	pass

Operators (both). Conv×45, MatMul×152, Gemm×5 (the quantizable compute), plus LayerNormalization×40, Sigmoid×59, Softmax×26, Erf×14 (GELU), GridSample×12, TopK×3, GatherElements×4, Einsum×1, and 954 Constant nodes (foldable). Because the file is ~98–99% FLOAT32 weights, halving weight width (FP16) roughly halves the file and INT8 roughly quarters the weight portion.

---

5. Variant matrix & sizes

All variants pass the ONNX checker and preserve the exact I/O contract of §3.

Variant	What it does	ECSeg-S		ECSeg-M
		MiB	% fp32	MiB	% fp32
fp32	baseline	39.95	100%	77.08	100%
graphopt	onnxslim: constant-fold, dead-node elimination, safe fusions (runtime-agnostic)	40.72	102%	77.85	101%
fp16	float16 weights, I/O kept float32 (boundary casts); conservative op block-list	20.57	51%	39.13	51%
int8-dynamic	weight-only dynamic INT8 (per-channel) on MatMul/Gemm	16.57	41%	32.75	42%
int8-static	static QDQ INT8 (Conv+MatMul), COCO calibration	12.87	32%	22.40	29%
int8-static-selective	static QDQ INT8 but mask-head nodes kept float	13.98	35%	24.41	32%

Note: graphopt is slightly larger — onnxslim folds Constant nodes into initializers, materializing a few tensors. Its only potential benefit is faster session-create, not smaller storage (the app already applies graphOptimizationLevel:'all' at load, so the runtime graph is equivalent either way).

---

6. Results

6.1 Loads-and-runs in onnxruntime-web/WASM (the gate)

Single-thread WASM (wasm-st): every variant loads and runs. All six S variants and six M variants create a session and produce the four outputs. Cross-runtime check: the FP32 baseline returns the same 17 instances on the reference image as Python onnxruntime — the harness is faithful.

Multi-thread WASM (wasm-mt, the cross-origin-isolated browser config): the ECSeg graph hard-hangs at session.create — for every variant, and for FP32 too. This was isolated carefully:

• Reproduced across every threaded configuration tried: numThreads auto(0) / 4 / 2, executionMode both parallel and sequential, both headless and headed, both chromium.launch and launchPersistentContext. All hang at session creation; the hang is a hard, synchronous block of the renderer (an in-page 30 s timeout never fires), i.e. not merely slow.
• It is ECSeg-graph-specific, not a harness artifact. The repo's existing scripts/benchmark-sam2.mjs harness — same onnxruntime-web 1.24.3, same single-file ORT bundle, same COOP/COEP server, same machine — brings the threaded WASM pool up fine and completes wasm-mt for SAM tiny/small/large. The ECSeg graph is what the threaded build cannot instantiate.
• This is the same class of failure the app already documents for ECSeg on the WebGPU EP (hence the hard executionProviders:['cpu'] pin): the graph's deformable-attention machinery — 12 GridSample ops, GatherElements, Einsum, dynamic TopK — is not handled by these ORT-web backends.

Consequences for this study.

1. The benchmark measures single-thread WASM, which is not a fallback but the config the app actually ships on Desktop (Tauri) and iOS (both pinned to one thread by session-parameters.ts), i.e. 2 of 3 platforms. It is also the cleaner per-op comparison.
2. For the pure browser (app.annotateit.ai, isolated → numThreads:0), ECSeg would request exactly the threaded pool that hangs here. Whether the app's bundler wiring avoids the hang in production is out of scope and untested — but optimization does not change it (every variant retains the GridSample machinery and hangs identically), so it is orthogonal to the swap decision.
3. Latency is input-independent for this architecture (static 640×640 input, fixed 300 queries, fixed 160×160 masks, no data-dependent loops), so a single reference input is representative for timing.

6.2 Browser latency (onnxruntime-web 1.24.3, CPU/WASM single-thread)

Apple M4 Max, Chrome, wasm-st (single thread — the config Desktop+iOS ship, and the only one the ECSeg graph runs in; see §6.1). 15 warm repeats after 2 warm-ups, identical input. Latency is input-independent for this static graph.

ECSeg-S (baseline warm p50 = 1299 ms)

variant	session-create ms	cold ms	warm p50 ms	p90	p95	Δ warm vs fp32	instances
fp32	529	1368	1299	1306	—	—	17
graphopt	435	1367	1300	1308	—	+0.1%	17
fp16	703	1437	1366	1377	—	+5.2%	17
int8-dynamic	—	—	does not load	—	—	—	—
int8-static	691	1625	1523	1535	—	+17.2%	0
int8-static-selective	683	1604	1493	1500	—	+14.9%	0

ECSeg-M (baseline warm p50 = 2093 ms)

variant	session-create ms	cold ms	warm p50 ms	p90	p95	Δ warm vs fp32	instances
fp32	531	2140	2093	2103	—	—	18
graphopt	434	2135	2089	2093	—	−0.2%	18
fp16	693	2238	2177	2185	—	+4.0%	18
int8-dynamic	—	—	does not load	—	—	—	—
int8-static	695	2482	2378	2388	—	+13.6%	0
int8-static-selective	673	2453	2343	2349	—	+11.9%	0

Reading the table.

• graphopt: warm latency identical; only a ~90 ms cheaper session-create; larger on disk. No win.
• fp16: slower in WASM (+4–5%). The WASM CPU EP has no native fp16 compute kernels, so ORT inserts cast-to-fp32 around every fp16 op — smaller weights, more work. S breaches the ≤5% warm budget.
• int8-dynamic: fails session.create in ort-web 1.24.3 — [ShapeInferenceError] Incompatible dimensions (ONNX-checker-valid but runtime-rejected). Hard fail.
• int8-static / -selective: slower (+12–17%: QuantizeLinear/DequantizeLinear overhead with no optimized INT8 kernels for this graph outweighs the smaller weights) and produce 0 instances — the head quantization collapses scores below the 0.4 threshold (calibration hit a zero-range weight in the segmentation head → zero scale → degenerate output; see §6.3). Keeping the mask head float (-selective) does not recover detection — the score/box heads are also destroyed.

Net: the two variants that reduce storage the most (INT8) are the worst on both latency and quality; the one with acceptable quality (fp16) is slower; the only lossless one (graphopt) is larger. No variant is simultaneously smaller, no-slower, and quality-neutral.

6.3 Numerical equivalence vs FP32

Candidate vs FP32 on identical inputs (Python onnxruntime CPU), 47 held-out datasets/ images, 358 (S) / ~360 (M) FP32 instances total. mean mask IoU is over matched instances; worst-img and min-inst expose the tail. (label_agreement_all300 is omitted — it is dominated by sub-threshold junk queries and is not meaningful; class match on real detections is 1.00 for every runnable variant.)

ECSeg-S

variant	cand/base inst	mean mask IoU	worst-img mask IoU	min-inst mask IoU	mask pixel flips	near-0.4 flips	NaN/Inf
graphopt	358/358	1.000	1.000	1.000	0.0%	0	no
fp16	357/358	0.988	0.667	0.000	13.9%	1	no
fp16-hi	357/358	0.985	0.667	0.000	14.0%	1	no
int8-dynamic	—	— (fails to load in ORT-web)
int8-static	0/358	n/a	n/a	n/a	—	358	no
int8-static-selective	0/358	n/a	n/a	n/a	—	358	no

ECSeg-M (341 FP32 instances total)

variant	cand/base inst	mean mask IoU	worst-img mask IoU	min-inst mask IoU	mask pixel flips	near-0.4 flips	NaN/Inf
graphopt	341/341	1.000	1.000	1.000	0.0%	0	no
fp16	344/341	0.979	0.613	0.000	13.5%	5	no
fp16-hi	342/341	0.976	0.548	0.000	13.3%	5	no
int8-dynamic	—	— (fails to load in ORT-web and Python ORT)
int8-static	0/341	n/a	n/a	n/a	—	341	no
int8-static-selective	0/341	n/a	n/a	n/a	—	341	no

M fp16 quality is worse than S (mean 0.976–0.979 vs 0.985–0.988; worst-image 0.55–0.61 vs 0.67) — the larger model's larger mask logits (abs error up to ~670) degrade more under fp16.

Reading it.

• graphopt is bit-identical to FP32 (as expected from a lossless graph rewrite).
• fp16 / fp16-hi: mean mask IoU ~0.985–0.988 looks high, but the tail is real — at least one instance drops to IoU 0.0 (a fully wrong mask) and the worst image averages 0.67. Raising the fp16 range clamp from the library default ±1e4 to fp16's true ±65504 (fp16-hi) does not help — the error is inherent fp16 precision loss in the large mask logits, not clipping (max mask-logit error stays ~300). For an annotation tool this occasional fully-wrong mask is a real UX cost.
• int8-static / -selective: 0 instances — all detections drop below the 0.4 score threshold (358 near-threshold decision flips; no NaN/Inf — the head simply collapses). Static INT8 calibration hit a zero-range weight in the segmentation head (zero scale → degenerate quantized output). Excluding the mask head (-selective) does not recover it — the score/box heads are equally affected.
• int8-dynamic: does not load in either runtime — ort-web 1.24.3 rejects it with [ShapeInferenceError] Incompatible dimensions and Python onnxruntime 1.22 fails to load the model at all. Per-channel dynamic quantization of this graph's MatMuls produces a structurally invalid graph (ONNX-checker-valid, yet unloadable) — a concrete instance of "the checker accepting it is not sufficient".

6.3.1 Side-by-side visualizations

Rendered by scripts/model-optimization/visualize.py (FP32 vs candidate, plus a union-mask disagreement panel). Stored under img/:

• ecseg-s-int8static-empty.png — the failure that matters most: FP32 finds 18 instances on a parking-lot image, int8-static finds 0 (union mask IoU 0.000). This is what "smaller" costs.
• ecseg-s-fp16-worstcase.png — FP32 vs fp16 on the worst held-out image for fp16 (per-instance mean mask IoU 0.667). Overall coverage is close (union IoU high), but individual instance boundaries drift — the per-instance metric is where fp16 loses.
• ecseg-m-fp16-typical.png — a typical image where M fp16 matches FP32 (union IoU 1.0). fp16's damage is concentrated on a minority of images, not uniform.

6.4 Cross-runtime & cross-browser correctness

The FP32 baseline and the lossless graphopt return the same instances in ORT-web/WASM as in Python onnxruntime (17 on the S reference image, 18 on M), confirming the harness is faithful and the WASM CPU EP matches the reference. The broken INT8 variants return 0 instances in both runtimes — a consistent (if useless) result, not a runtime discrepancy.

Cross-browser (WebKit / Safari engine, single-thread WASM, ECSeg-S). Every Chrome finding reproduces on WebKit — the conclusions are not a Blink quirk:

variant	WebKit warm p50	instances	vs Chrome
fp32	1360 ms	17	same behavior (Chrome 1299 ms)
fp16	1455 ms (+7% vs its own fp32)	17	slower, same as Chrome
int8-dynamic	does not load (session.create fails)	—	same as Chrome
int8-static	1704 ms (+25%)	0	slower + empty, same as Chrome

---

7. Limitations

• Quality proxy, not full COCO AP. The decision metric is agreement-vs-FP32 (FP32 is the reference for a drop-in replacement) on a 47-image held-out set, plus a 50-image COCO calibration set. True mask/box AP against COCO ground truth (with pycocotools) is not computed here; the quality budget (≤0.5 AP drop) is therefore evaluated via strict proxies (mask IoU vs FP32, near-threshold decision flips), and any verdict resting on the proxy is marked accordingly.
• One machine, two browser engines. All numbers are Apple M4 Max; Chrome (Blink) is primary and WebKit (Safari engine) confirms the four decisive S cases (§6.4). No Windows / Android / low-end-device numbers — but the verdict rests on relative deltas (smaller-but- slower, won't-load, 0-instances) that are architectural, not device-specific, so they are unlikely to invert on other hardware.
• Multi-thread WASM could not be measured for the ECSeg graph (it hard-hangs session.create); see §6.1. Single-thread is the config that runs, and is what Desktop + iOS ship. Whether the pure-browser multi-thread path works in production is a separate, pre-existing question independent of this optimization decision.
• Because no variant qualified, no adoption/rollback path was exercised. The registry (custom-checkpoints.ts), IndexedDB save/load, curated-import verification, Test flow and real inference were left on FP32 (the ECSeg jest suites — 63 tests — still pass; repo type-check is green).

---

8. Verdicts

ECSeg-S — REJECT

No variant is simultaneously smaller, no-slower, and quality-neutral in the target runtime:

candidate	size	warm latency	quality	loads/runs	why not
graphopt	+2% (larger)	≈ equal	bit-exact	yes	no size or latency win; only ~90 ms cheaper create
fp16 / fp16-hi	51%	+5.2% slower	mask IoU 0.985–0.988 mean but 0.0 min-inst / 0.67 worst-img, 14% flips	yes	slower and occasional fully-wrong masks; over the 5% budget
int8-dynamic	41%	—	—	NO (ShapeInferenceError)	does not create a session in ort-web 1.24.3
int8-static	32%	+17% slower	0 instances	yes (but useless)	head collapses; slower too
int8-static-selective	35%	+15% slower	0 instances	yes (but useless)	head collapses; slower too

Keep the FP32 checkpoint. The only lossless option is larger; every smaller option is slower and/or broken.

ECSeg-M — REJECT

Same picture, and fp16's quality is worse on M than on S (larger logits degrade more under fp16):

candidate	size	warm latency	quality	loads/runs	why not
graphopt	+1% (larger)	≈ equal	bit-exact	yes	no win
fp16 / fp16-hi	49%	+4.0% slower	mask IoU 0.976 mean but 0.0 min-inst / 0.55 worst-img, 13% flips	yes	slower and worse masks than S; a segmentation-quality regression the product should not ship for a size-only gain
int8-dynamic	42%	—	—	NO (ShapeInferenceError)	does not create a session
int8-static	29%	+14% slower	0 instances	yes (but useless)	head collapses; slower too
int8-static-selective	32%	+12% slower	0 instances	yes (but useless)	head collapses; slower too

M fp16 sits just inside the 5% latency budget, so it is the single "closest" candidate — but it buys a 49% storage cut with a latency regression and a measurable mask-quality regression (worst-image mask IoU 0.55, occasional IoU-0 instances), and AnnotateIt weights annotation quality and latency above compression. Not worth adopting, even as a device-conditional OPTIONAL. Keep the FP32 checkpoint.

Why every path failed — root cause

The ECSeg head is a DETR-style deformable-attention segmentation head (12 GridSample, Einsum, dynamic TopK, GatherElements). Three independent facts fall out of that one design:

1. It won't run on the threaded WASM pool or WebGPU — only single-thread CPU/WASM. (Already why the app pins ['cpu'].)
2. Its mask logits have a wide dynamic range (errors up to ~670 in M), so fp16 loses real precision exactly where it matters — near the mask-logit=0 decision boundary — flipping pixels.
3. Its score/box heads have low-rank / near-constant weight tensors, so static-INT8 MinMax calibration produces zero-scale quantizers that collapse the output to nothing.

And because the graph is ~98% weights with almost no INT8-friendly large-matmul structure that ort-web has fast kernels for, the QDQ/cast overhead makes the quantized graphs slower, not faster, on WASM. Smaller file, more runtime work: the exact anti-pattern the study was asked to rule out.

---

9. Reproduce

See scripts/model-optimization/README.md for the exact commands (download+verify, build variants, correctness, browser benchmark, aggregate).