Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,121 @@
+---
+license: apache-2.0
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- image-segmentation
+- zero-shot-image-classification
+base_model:
+- CIDAS/clipseg-rd64-refined
+---
+# CLIPSeg (rd64-refined) — ExecuTorch (segmentation from a text prompt)
+
+Say what you want and get a mask for it. The shelf's other open-vocabulary model,
+Grounding DINO, returns **boxes** for a phrase; SAM and EdgeTAM return a **mask for a
+click**. This returns a **mask for a phrase**, which neither of those does.
+
+Two files, because the expensive half does not depend on the prompt:
+
+```
+clipseg_vision   pixel_values (1, 3, 352, 352) fp32
+    -> three activations, each (1, 485, 768)     # CLIP ViT-B/16 layers 3, 6, 9
+clipseg_segment  a3, a6, a9, input_ids (1, 77) int64
+    -> logits (1, 352, 352) fp32                 # per-pixel; sigmoid for 0..1
+```
+
+- **Source**: [CIDAS/clipseg-rd64-refined](https://huggingface.co/CIDAS/clipseg-rd64-refined) — 150.7M parameters
+- **License**: apache-2.0
+- **Threshold**: `logits > 0` is the mask. Everything below is background.
+
+**Run the vision half once per image and the segment half once per phrase.** That is the
+whole point of the split: trying "a tree", "the sky" and "grass" on one photograph costs
+one ViT pass and three cheap ones, not three ViT passes. The three activations are 4.5 MB
+in total — an intermediate the caller holds, not a file.
+
+Checked against upstream's single-shot `forward`: **max_abs_diff 0.000e+00** on every
+prompt. Splitting it changes nothing about the arithmetic.
+
+## Variants
+
+| half | build | file | size (MB) | Mac ms* |
+|---|---|---|---|---|
+| vision | fp32 | `clipseg_vision_xnnpack_fp32.pte` | 286.7 | 194.8 |
+| vision | fp16 | `clipseg_vision_xnnpack_fp16.pte` | 173.1 | 252.4 |
+| segment | fp32 | `clipseg_segment_xnnpack_fp32.pte` | 260.8 | 24.7 |
+| segment | fp16 | `clipseg_segment_xnnpack_fp16.pte` | 129.6 | 41.8 |
+| segment | **Core ML (fp16, iOS)** | `clipseg_segment_coreml_all.pte` | 132.2 | **3.2** |
+
+\*Mac arm64, median of 10 — a reference point for relative cost, not a device number.
+Torch eager fp32 on the same machine is 53 ms for the vision half and 19 ms for the
+segment half.
+
+**The vision half is slower than eager on XNNPACK and there is no Core ML build for it.**
+Both are worth knowing before you plan around this model:
+
+- Core ML refuses a graph holding any non-contiguous `_clone_dim_order` node
+  (`AssertionError: Only contiguous memory format is supported in CoreML`). The vision
+  half has two — ExecuTorch's dim-order pass puts the patch convolution's output in
+  channels-last, and `interpolate_pos_encoding` produces another. Rewriting the patch
+  convolution as the equivalent linear was tried and only moved the problem: the
+  reshape-and-permute that gathers patches earns a channels-last clone of its own. The
+  segment half has no 4-D convolution and lowers to Core ML whole, 100% delegated.
+- On XNNPACK the vision half delegates 66.8% of its ops across 43 subgraphs. **Why it is
+  3.7x slower than eager has not been measured**, so this card does not name a cause.
+
+## Verification
+
+Correlation over a mask is not the gate. A mask gets used by thresholding it, so the
+number that matters is how many pixels change side:
+
+| build | worst mask IoU vs eager |
+|---|---|
+| fp32 | **1.0000** |
+| fp16 | 0.9974 |
+| Core ML segment | 0.9994 |
+| int8 | 0.9429 — **withheld** |
+
+**Only prompts the reference actually answers are scored.** On the gate image the model
+answers "a tree" (63% of pixels), "the sky" (21%) and "grass" (5%), and ignores "a road"
+(0%) — an empty mask agrees with another empty mask perfectly, so scoring the ignored
+prompt would flatter every build. The four prompts were picked by running the reference
+over a dozen candidates rather than written from imagination.
+
+## Running it
+
+**1. Preprocess to 352x352**, with `CLIPSegProcessor`. The CLIP backbone was trained at
+224 and CLIPSeg runs it at 352 by interpolating the position embeddings — feeding it 224
+gives a mask at the wrong scale rather than an error.
+
+**2. Tokenise the prompt to a fixed 77**, CLIP's context length:
+
+```python
+ids = processor.tokenizer([prompt], padding="max_length", truncation=True,
+                          max_length=77, return_tensors="pt")["input_ids"]
+```
+
+Padding to the full 77 is safe here and it is worth knowing why: CLIP's text encoder is
+causal and pools at the end-of-text token, so the padding that follows never reaches it.
+Measured against the processor's own tight padding: **6.676e-06**.
+
+**3. Vision once, segment per prompt**, then threshold at 0 and resize the 352x352 mask
+back to your image.
+
+## Not shipped: int8
+
+Both halves are withheld together, because the gate measures the pair. `clipseg_vision`
+would come out at **83.5 MB** from 286.7 (dynamic int8 has almost nothing to leave alone
+in a ViT — there is no token embedding table), and `clipseg_segment` at **141.6 MB**,
+which is *larger* than its own fp16 at 129.6 MB: the segment half carries CLIP's
+49,408 x 512 text embedding table, 101 MB of its 260.8, a 38.8% share. This shelf's rule
+`int8/fp16 = 0.5 + 1.5 x (table share)` predicts 1.082 and it came out at 1.092.
+
+The quality is what decides it. Worst mask IoU **0.9429** against a 0.95 bar, and the
+worst case is the *smallest* mask — "grass", 5% of the image, where boundary pixels are
+most of the mask. Correlation for the same build reads 0.9967 and 0.9999 on the two
+halves, which no correlation gate would stop.
+
+This repo holds **both towers**: `clip_vit_b32_image_xnnpack_fp32.pte` (image) and
+`clip_vit_b32_text_xnnpack_fp32.pte` (text, fixed len 77 + attention mask).
+L2-normalize both embeddings, then cosine-match.