README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- image-segmentation
- zero-shot-image-classification
base_model:
- CIDAS/clipseg-rd64-refined
---
# CLIPSeg (rd64-refined) — ExecuTorch (segmentation from a text prompt)

Say what you want and get a mask for it. The shelf's other open-vocabulary model,
Grounding DINO, returns **boxes** for a phrase; SAM and EdgeTAM return a **mask for a
click**. This returns a **mask for a phrase**, which neither of those does.

Two files, because the expensive half does not depend on the prompt:

```
clipseg_vision   pixel_values (1, 3, 352, 352) fp32
    -> three activations, each (1, 485, 768)     # CLIP ViT-B/16 layers 3, 6, 9
clipseg_segment  a3, a6, a9, input_ids (1, 77) int64
    -> logits (1, 352, 352) fp32                 # per-pixel; sigmoid for 0..1
```

- **Source**: [CIDAS/clipseg-rd64-refined](https://huggingface.co/CIDAS/clipseg-rd64-refined) — 150.7M parameters
- **License**: apache-2.0
- **Threshold**: `logits > 0` is the mask. Everything below is background.

**Run the vision half once per image and the segment half once per phrase.** That is the
whole point of the split: trying "a tree", "the sky" and "grass" on one photograph costs
one ViT pass and three cheap ones, not three ViT passes. The three activations are 4.5 MB
in total — an intermediate the caller holds, not a file.

Checked against upstream's single-shot `forward`: **max_abs_diff 0.000e+00** on every
prompt. Splitting it changes nothing about the arithmetic.

## Variants

| half | build | file | size (MB) | Mac ms* |
|---|---|---|---|---|
| vision | fp32 | `clipseg_vision_xnnpack_fp32.pte` | 286.7 | 194.8 |
| vision | fp16 | `clipseg_vision_xnnpack_fp16.pte` | 173.1 | 252.4 |
| segment | fp32 | `clipseg_segment_xnnpack_fp32.pte` | 260.8 | 24.7 |
| segment | fp16 | `clipseg_segment_xnnpack_fp16.pte` | 129.6 | 41.8 |
| segment | **Core ML (fp16, iOS)** | `clipseg_segment_coreml_all.pte` | 132.2 | **3.2** |

\*Mac arm64, median of 10 — a reference point for relative cost, not a device number.
Torch eager fp32 on the same machine is 53 ms for the vision half and 19 ms for the
segment half.

**The vision half is slower than eager on XNNPACK and there is no Core ML build for it.**
Both are worth knowing before you plan around this model:

- Core ML refuses a graph holding any non-contiguous `_clone_dim_order` node
  (`AssertionError: Only contiguous memory format is supported in CoreML`). The vision
  half has two — ExecuTorch's dim-order pass puts the patch convolution's output in
  channels-last, and `interpolate_pos_encoding` produces another. Rewriting the patch
  convolution as the equivalent linear was tried and only moved the problem: the
  reshape-and-permute that gathers patches earns a channels-last clone of its own. The
  segment half has no 4-D convolution and lowers to Core ML whole, 100% delegated.
- On XNNPACK the vision half delegates 66.8% of its ops across 43 subgraphs. **Why it is
  3.7x slower than eager has not been measured**, so this card does not name a cause.

## Verification

Correlation over a mask is not the gate. A mask gets used by thresholding it, so the
number that matters is how many pixels change side:

| build | worst mask IoU vs eager |
|---|---|
| fp32 | **1.0000** |
| fp16 | 0.9974 |
| Core ML segment | 0.9994 |
| int8 | 0.9429 — **withheld** |

**Only prompts the reference actually answers are scored.** On the gate image the model
answers "a tree" (63% of pixels), "the sky" (21%) and "grass" (5%), and ignores "a road"
(0%) — an empty mask agrees with another empty mask perfectly, so scoring the ignored
prompt would flatter every build. The four prompts were picked by running the reference
over a dozen candidates rather than written from imagination.

## Running it

**1. Preprocess to 352x352**, with `CLIPSegProcessor`. The CLIP backbone was trained at
224 and CLIPSeg runs it at 352 by interpolating the position embeddings — feeding it 224
gives a mask at the wrong scale rather than an error.

**2. Tokenise the prompt to a fixed 77**, CLIP's context length:

```python
ids = processor.tokenizer([prompt], padding="max_length", truncation=True,
                          max_length=77, return_tensors="pt")["input_ids"]
```

Padding to the full 77 is safe here and it is worth knowing why: CLIP's text encoder is
causal and pools at the end-of-text token, so the padding that follows never reaches it.
Measured against the processor's own tight padding: **6.676e-06**.

**3. Vision once, segment per prompt**, then threshold at 0 and resize the 352x352 mask
back to your image.

## Not shipped: int8

Both halves are withheld together, because the gate measures the pair. `clipseg_vision`
would come out at **83.5 MB** from 286.7 (dynamic int8 has almost nothing to leave alone
in a ViT — there is no token embedding table), and `clipseg_segment` at **141.6 MB**,
which is *larger* than its own fp16 at 129.6 MB: the segment half carries CLIP's
49,408 x 512 text embedding table, 101 MB of its 260.8, a 38.8% share. This shelf's rule
`int8/fp16 = 0.5 + 1.5 x (table share)` predicts 1.082 and it came out at 1.092.

The quality is what decides it. Worst mask IoU **0.9429** against a 0.95 bar, and the
worst case is the *smallest* mask — "grass", 5% of the image, where boundary pixels are
most of the mask. Correlation for the same build reads 0.9967 and 0.9999 on the two
halves, which no correlation gate would stop.

This repo holds **both towers**: `clip_vit_b32_image_xnnpack_fp32.pte` (image) and
`clip_vit_b32_text_xnnpack_fp32.pte` (text, fixed len 77 + attention mask).
L2-normalize both embeddings, then cosine-match.