README.md

---
license: mit
library_name: litert
pipeline_tag: image-segmentation
tags:
  - litert
  - tflite
  - android
  - on-device
  - gpu
  - instance-segmentation
  - yolact
  - coco
  - real-time
---

# YOLACT-ResNet50 — LiteRT (real-time instance segmentation, GPU)

On-device **real-time instance segmentation** running **fully on the LiteRT
`CompiledModel` GPU** delegate (no CPU fallback). [YOLACT](https://arxiv.org/abs/1904.02689)
(ICCV 2019) predicts per-instance COCO masks. The network (ResNet50 + FPN +
protonet + heads) runs on the GPU; the lightweight decode (NMS + linear-combination
masks) runs host-side. ~41 ms/graph on a Pixel 8a.

- **Architecture:** YOLACT-ResNet50 (base, no deformable conv) — pure CNN.
- **Weights:** [dbolya/yolact](https://github.com/dbolya/yolact) (`yolact_resnet50_54_800000`) · MIT.
- **Size:** 125 MB.

![YOLACT instance segmentation](hero.png)

## Files

- `yolact.tflite` — the GPU graph (input `[1,3,550,550]` NCHW).
- `priors.bin` — 19248 SSD priors `[cx,cy,w,h]` (float32) used by the host-side box decode.

## I/O

- **Input:** `[1, 3, 550, 550]` NCHW, **BGR**, normalized `(x - [103.94,116.78,123.68]) / [57.38,57.12,58.40]`
  (no /255).
- **Raw outputs:** `loc [1,19248,4]`, `conf [1,19248,81]` (softmax, incl. background),
  `mask [1,19248,32]` (coefficients), `proto [1,138,138,32]` (prototype masks).

## Host-side decode

1. **Boxes:** SSD `decode(loc, priors, variances=[0.1,0.2])`.
2. **NMS:** per-class, score-threshold ~0.3, IoU 0.5, top-k.
3. **Masks (lincomb):** for each kept detection, `mask = sigmoid(proto @ coeff)` → crop
   to the box → threshold 0.5 → upscale.

## GPU conversion

Base YOLACT is a pure CNN, so the graph converts fully GPU-compatible (**138/138
nodes on the delegate, 1 partition**; device corr 0.99999–1.0 vs PyTorch on all four
raw outputs) with **one patch**: the ResNet50 stem `MaxPool2d(padding=1)` lowers to a
`-inf` PADV2 (rejected by Mali), replaced by a 0-pad + unpadded maxpool (exact
post-ReLU). The scripted FPN is made traceable by disabling YOLACT's JIT
(`use_jit=False`). CPU-exact vs PyTorch (corr 1.0).

## Minimal usage

### Kotlin (Android, LiteRT CompiledModel GPU)

```kotlin
val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "yolact.tflite", options, null)
val inBufs = model.createInputBuffers()
val outBufs = model.createOutputBuffers()   // map by size: loc=N*4, conf=N*81, mask=N*32, proto=138*138*32

inBufs[0].writeFloat(inputNCHW)              // [1,3,550,550] BGR, (x-[103.94,116.78,123.68])/[57.38,57.12,58.40]
model.run(inBufs, outBufs)
val loc = outBufs[iLoc].readFloat()          // [19248*4]
val conf = outBufs[iConf].readFloat()        // [19248*81] (softmax)
val mask = outBufs[iMask].readFloat()        // [19248*32] coefficients
val proto = outBufs[iProto].readFloat()      // [138*138*32] prototypes

// host-side decode (priors.bin bundled as an asset):
//   box = SSD-decode(loc, priors, variances=[0.1,0.2]); per-class NMS (score 0.3, IoU 0.5);
//   per kept det: mask = sigmoid(proto @ coeff) (>0) cropped to the box.
// Full implementation: YolactSegmenter.kt in the sample app.
```

### Python (LiteRT / ai-edge-litert)

```python
import numpy as np
from ai_edge_litert.interpreter import Interpreter

it = Interpreter(model_path="yolact.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()
it.set_tensor(inp[0]["index"], x)          # [1,3,550,550] BGR, normalized (see above)
it.invoke()
outs = {tuple(o["shape"][1:]): it.get_tensor(o["index"])[0] for o in out}
loc  = outs[(19248, 4)]; conf = outs[(19248, 81)]
mask = outs[(19248, 32)]; proto = outs[(138, 138, 32)]

priors = np.fromfile("priors.bin", np.float32).reshape(-1, 4)
cxy = priors[:, :2] + loc[:, :2] * 0.1 * priors[:, 2:]
wh  = priors[:, 2:] * np.exp(loc[:, 2:] * 0.2)
boxes = np.concatenate([cxy - wh / 2, cxy + wh / 2], 1)      # x1y1x2y2 (0..1)
# then per-class NMS on conf, and mask_i = sigmoid(proto @ mask[i]) cropped to boxes[i]
```

## License

MIT (YOLACT / dbolya/yolact). COCO class taxonomy.