Remove backend experiment notes from the published model repository

README.md

@@ -48,15 +48,13 @@ These measure export agreement, not accuracy against ground-truth masks.
 Choose FP32 when boundary precision matters.
 
 Full-network INT8 remains unpublished because it loses substantial foreground
-detail. See [XNNPACK precision notes](xnnpack/NOTES.md) for the selective recipe,
-calibration, validation and rejected alternatives.
+detail.
 
 Vulkan FP16 preserves input normalization and one sensitive convolution in FP32.
 On a Galaxy S20+ (Android 13, Mali), native forward medians were 337 ms for this
 variant, versus 316 ms for XNNPACK FP32 and 189 ms for selective INT8. Android uses the standard Vulkan-first default when the Vulkan backend is
 linked. Performance varies by device. Across 13 device-tested images, Vulkan FP16 averaged 0.9945 mask IoU
-against FP32 (minimum 0.9854). See [Vulkan precision notes](vulkan/NOTES.md)
-for the export recipe, phone validation and timing limitations.
+against FP32 (minimum 0.9854).
 
 ## Repository structure
 
@@ -66,10 +64,8 @@ coreml/config.json                           1.5 kB
 coreml/u2netp_coreml_fp16.pte                3.3 MB
 coreml/u2netp_coreml_fp32.pte                5.5 MB
 vulkan/config.json                            886 B
-vulkan/NOTES.md                              4.1 kB
 vulkan/u2netp_vulkan_fp16.pte                2.4 MB
 xnnpack/config.json                          1.5 kB
-xnnpack/NOTES.md                             6.6 kB
 xnnpack/u2netp_xnnpack_fp32.pte              4.6 MB
 xnnpack/u2netp_xnnpack_int8.pte              4.0 MB
 LICENSE                                     11.4 kB

vulkan/NOTES.md

@@ -1,90 +0,0 @@
-# Vulkan precision and Android validation
-
-The published FP16 variant accepts RGB float32 `[1,3,320,320]` in `[0,1]` and returns float32
-`[1,1,320,320]` foreground logits. Input normalization stays in FP32. The
-export contains one Vulkan delegate for the network and one portable input
-`amax` reduction. Texture limits are capped at 2048 in each dimension.
-
-## FP16 with a sensitive FP32 convolution
-
-The FP16 export fuses Conv/BatchNorm pairs before casting the network to half.
-`stage2d.rebnconv1d.conv_s1` retains FP32 weights, bias and computation, with
-casts around it inside the Vulkan delegate. Global `force_fp16` is deliberately
-not used: it would also downcast this convolution and input normalization.
-
-On the connected Galaxy S20+, an entirely FP16 export changed the bicycle mask
-to 0.9418 IoU against XNNPACK FP32. Keeping this convolution in FP32 recovered
-0.9932 IoU. The entirely FP16 candidate is not published.
-
-## Device checks
-
-Validation used Samsung Galaxy S20+ SM-G986B, Android 13, Mali GPU, arm64-v8a,
-and the ExecuTorch 1.4.1 runtime/backend libraries shipped with RNE. A native
-runner linked these exact libraries, loaded each actual `.pte`, checked float32
-output type and element count, rejected non-finite output, and saved logits.
-Exported method metadata was also checked against the config.
-
-The comparison set is the same 13 images used for selective XNNPACK INT8
-validation: `bike.jpg`, `boat.jpg`, `whisk.png`, plus COCO val2017 image IDs 139,
-285, 632, 724, 776, 785, 802, 872, 885 and 1000. All-black and all-white inputs
-were also checked. These are agreement measurements against the shipped
-XNNPACK FP32 `.pte`, not ground-truth segmentation accuracy.
-
-| Variant | Mean mask IoU (13 images) | Minimum mask IoU | File size |
-| --- | --- | --- | --- |
-| Vulkan FP16 with one FP32 convolution | 0.9945 | 0.9854 | 2,437,516 B |
-
-The FP16 probability MAE was at most 0.00444 across those 13 images. The published
-variant produced finite outputs on all 15 inputs.
-
-Warm native forward measurements on the same bicycle input used three warm-up
-runs followed by ten measured runs per variant:
-
-| Variant | Median |
-| --- | --- |
-| XNNPACK FP32 | 315.6 ms |
-| XNNPACK selective INT8 | 189.1 ms |
-| Vulkan FP16 with one FP32 convolution | 337.2 ms |
-| Vulkan FP32 (unpublished trial) | 403.1 ms |
-
-These were sequential host-driven device checks, not interleaved benchmarks;
-they exclude model loading, image preprocessing and rendering. Vulkan does not
-win on this older Mali GPU. Android uses the standard Vulkan-first default
-when the Vulkan backend is linked. Select XNNPACK INT8 for the faster option
-on this tested phone; performance varies by device. A trial with only four half-precision convolutions was
-also slower (421.9 ms) and is not published.
-
-## Reproducing the export environment
-
-The Vulkan AOT implementation is pinned to software-mansion-labs/executorch
-`@ms/separate-backends-1.4.1`, commit
-`0de5ce9470e4a18d6d45585f69519267bbe218b9`. Torch is 2.14.0. The nested
-`pyproject.toml` records the fork dependency; see the repository Vulkan skill
-for the source-built environment used by other Vulkan models.
-
-For AOT-only export, a compatible ExecuTorch 1.4.1 wheel can provide the common
-serialization binaries while the fork supplies its Vulkan Python package.
-No host GPU runtime is needed; Android validation still uses the shipped native
-Vulkan library. Clone the fork into a directory named `executorch`, check out
-the commit above, then from export-scripts run:
-
-```bash
-uv run --project exporter_models/u2netp python - /path/to/executorch <<'PY'
-import runpy
-import sys
-from pathlib import Path
-import executorch.backends
-
-fork = Path(sys.argv[1]).resolve()
-executorch.backends.__path__.insert(0, str(fork / "backends"))
-sys.argv = [
-    "exporter", "export", "u2netp", "--backend", "vulkan", "--all-variants",
-    "--output-dir", "exports/u2netp/vulkan",
-]
-runpy.run_module("exporter.cli", run_name="__main__")
-PY
-```
-
-The release FP16 export was byte-identical to the device-validated candidate.
-The Python wheel's native runtime cannot execute Vulkan on this Mac; it was
-not used as a substitute for phone validation.

xnnpack/NOTES.md

@@ -1,119 +0,0 @@
-# XNNPACK precision experiments
-
-FP32 and selective INT8 are published. Full-network static INT8, an earlier
-mixed INT8/FP32 recipe and FP16 were rejected because they changed the masks.
-
-## Selective INT8 (published)
-
-The final recipe quantizes only `stage1d.rebnconvin`, `stage1d.rebnconv1d`,
-`stage2d.rebnconvin` and `stage2.rebnconvin`. These convolutions dominate host
-latency. Remaining convolutions, concatenations and residual additions stay
-FP32. Public I/O and normalization are unchanged.
-
-PT2E uses per-channel symmetric INT8 weights and per-tensor affine INT8
-activations with HistogramObserver. The weight observer minimum scale is
-float32 epsilon. Calibration uses the same 13 pinned upstream images listed
-below. Export scripts download these images automatically.
-
-ExecuTorch 1.4.1's concatenation annotator ignores its filter argument, so
-module exclusions alone do not keep skip connections in FP32. The export
-selects only convolution operators, with an explicit module filter. Weight
-attributes are allowed through this filter because they lack module-stack
-metadata. This avoids quantizing unrelated concatenations.
-
-Actual `.pte` execution was checked against the published FP32 `.pte` on the
-three upstream images below plus ten COCO val2017 images, IDs 139, 285, 632,
-724, 776, 785, 802, 872, 885 and 1000. None was used for calibration. Average
-mask IoU was 0.9953, minimum 0.9868; probability MAE ranged from 0.00015 to
-0.00543. All-black and all-white inputs also produced finite outputs.
-
-| Image       | Selective INT8 mask IoU | Probability MAE |
-| ----------- | ---------------------- | --------------- |
-| `bike.jpg`  | 0.9947                 | 0.00158         |
-| `boat.jpg`  | 0.9956                 | 0.00152         |
-| `whisk.png` | 0.9992                 | 0.00055         |
-
-On an Apple M1 Pro, 30 interleaved ExecuTorch runs after five warm-ups measured
-156.0 ms median for FP32 and 87.4 ms for selective INT8 (1.78× faster). File
-sizes were 4,610,948 and 4,007,684 bytes. These are host measurements, exclude
-RNE preprocessing/rendering, and need on-device confirmation. This small
-sample measures conversion agreement, not ground-truth segmentation accuracy.
-
-The event-traced ExecuTorch 1.4.1 runtime and cloned profiler confirmed all
-four selected convolutions execute as quantized XNNPACK kernels. Over five
-real-input executions, the delegate accounted for 99.36% of method time and
-the portable input `amax` for 0.63%.
-
-## Static INT8
-
-The experiment used PT2E `prepare_pt2e` / `convert_pt2e`,
-`XNNPACKQuantizer`, and `get_symmetric_quantization_config(is_per_channel=True)`:
-static INT8 activations with per-channel INT8 weights. Public inputs and
-outputs remained float32. The export included the same image-maximum and
-ImageNet normalization as FP32, and returned foreground logits before sigmoid.
-
-Calibration used 13 RGB images from the upstream repository at commit
-`ac7e1c817ecab7c7dff5ce6b1abba61cd213ff29`, resized to 320×320:
-`0002-01.jpg`, `0003.jpg`, `girl.png`, `hockey.png`, `horse.jpg`, `im_01.png`,
-`im_14.png`, `im_21.png`, `im_27.png`, `lamp2_meitu_1.jpg`, `long.jpg`,
-`rifle1.jpg`, and `sailboat3.jpg`.
-
-The three images below were held out from calibration. Results compare actual
-ExecuTorch `.pte` execution against the shipped FP32 `.pte`. Mask IoU uses a
-foreground-probability threshold of 0.5; MAE is measured after sigmoid in [0,1].
-These measure conversion agreement, not accuracy against ground-truth masks.
-
-| Image       | Static INT8 mask IoU | Static INT8 probability MAE | Mixed INT8 mask IoU | Mixed INT8 probability MAE |
-| ----------- | -------------------- | --------------------------- | ------------------- | -------------------------- |
-| `bike.jpg`  | 0.6692               | 0.09813                     | 0.9629              | 0.01421                    |
-| `boat.jpg`  | 0.8703               | 0.04655                     | 0.9410              | 0.02156                    |
-| `whisk.png` | 0.9387               | 0.06170                     | 0.9945              | 0.00458                    |
-
-Static INT8 is smaller and faster, but loses substantial foreground detail.
-Those mask changes outweigh the size and latency improvements.
-
-## Mixed INT8/FP32
-
-The earlier candidate excluded convolutions in `stage1`, `stage2`, `stage1d`,
-`stage2d`, `side1`, `side2`, and `outconv`, using the same calibration data.
-The concatenation-filter issue described above still quantized skip connections,
-including those inside excluded modules. Agreement improved, but the boat mask
-still fell to 0.9410 IoU and the latency gain was small. This candidate is not
-published. The selective recipe above replaces this approach.
-
-## Host size and latency
-
-Interleaved measurements used the same bicycle input, three warm-up calls,
-then 20 executions per candidate through ExecuTorch on an Apple M1 Pro running
-macOS 26.5.2, with torch 2.14.0, ExecuTorch 1.4.1 and torchao 0.18.0.
-
-| Candidate                  | File size | Median inference |
-| -------------------------- | --------- | ---------------- |
-| FP32 (published)           | 4.61 MB   | 217 ms           |
-| Static INT8 (rejected)     | 1.36 MB   | 64 ms            |
-| Mixed INT8/FP32 (rejected) | 2.61 MB   | 215 ms           |
-
-Other export/build work was running on the host, so these are indicative
-comparisons rather than isolated device benchmarks. They exclude RNE image
-preprocessing and rendering. The small calibration/validation sample does not
-prove that every quantization strategy is unsuitable; it establishes why these
-particular candidates were not shipped. A future attempt needs more calibration
-coverage and a larger independent quality evaluation before publication.
-
-## FP16 trial
-
-An XNNPACK FP16 candidate kept input normalization and public I/O in FP32 and
-fused each Conv/BatchNorm pair before converting the network to FP16. Eager
-agreement looked promising, but the actual exported runtime shifted the bicycle
-mask to 0.9549 IoU (probability MAE 0.01111) against the shipped FP32 model.
-It is not published either. This result is specific to the XNNPACK FP16
-candidate; the published CoreML FP16 artifact was validated separately.
-
-## Delegation
-
-A local event-traced ExecuTorch 1.4.1 build and `executorch-profiler` measured
-five FP32 executions: 99.68% of elapsed time was inside the XNNPACK delegate and
-0.31% was the portable `amax` input reduction. Convolution dominated execution.
-These are timing shares, not a count of delegated graph operators. Removing
-the reduction would not address the main cost and would change the upstream
-image-dependent input normalization.