litert-community/YOLACT-ResNet50-LiteRT

YOLACT-ResNet50 — LiteRT (real-time instance segmentation, GPU)

On-device real-time instance segmentation running fully on the LiteRT CompiledModel GPU delegate (no CPU fallback). YOLACT (ICCV 2019) predicts per-instance COCO masks. The network (ResNet50 + FPN + protonet + heads) runs on the GPU; the lightweight decode (NMS + linear-combination masks) runs host-side. ~41 ms/graph on a Pixel 8a.

• Architecture: YOLACT-ResNet50 (base, no deformable conv) — pure CNN.
• Weights: dbolya/yolact (yolact_resnet50_54_800000) · MIT.
• Size: 125 MB.

Files

• yolact.tflite — the GPU graph (input [1,3,550,550] NCHW).
• priors.bin — 19248 SSD priors [cx,cy,w,h] (float32) used by the host-side box decode.

I/O

• Input: [1, 3, 550, 550] NCHW, BGR, normalized (x - [103.94,116.78,123.68]) / [57.38,57.12,58.40] (no /255).
• Raw outputs: loc [1,19248,4], conf [1,19248,81] (softmax, incl. background), mask [1,19248,32] (coefficients), proto [1,138,138,32] (prototype masks).

Host-side decode

1. Boxes: SSD decode(loc, priors, variances=[0.1,0.2]).
2. NMS: per-class, score-threshold ~0.3, IoU 0.5, top-k.
3. Masks (lincomb): for each kept detection, mask = sigmoid(proto @ coeff) → crop to the box → threshold 0.5 → upscale.

GPU conversion

Base YOLACT is a pure CNN, so the graph converts fully GPU-compatible (138/138 nodes on the delegate, 1 partition; device corr 0.99999–1.0 vs PyTorch on all four raw outputs) with one patch: the ResNet50 stem MaxPool2d(padding=1) lowers to a -inf PADV2 (rejected by Mali), replaced by a 0-pad + unpadded maxpool (exact post-ReLU). The scripted FPN is made traceable by disabling YOLACT's JIT (use_jit=False). CPU-exact vs PyTorch (corr 1.0).

Minimal usage

Kotlin (Android, LiteRT CompiledModel GPU)

val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "yolact.tflite", options, null)
val inBufs = model.createInputBuffers()
val outBufs = model.createOutputBuffers()   // map by size: loc=N*4, conf=N*81, mask=N*32, proto=138*138*32

inBufs[0].writeFloat(inputNCHW)              // [1,3,550,550] BGR, (x-[103.94,116.78,123.68])/[57.38,57.12,58.40]
model.run(inBufs, outBufs)
val loc = outBufs[iLoc].readFloat()          // [19248*4]
val conf = outBufs[iConf].readFloat()        // [19248*81] (softmax)
val mask = outBufs[iMask].readFloat()        // [19248*32] coefficients
val proto = outBufs[iProto].readFloat()      // [138*138*32] prototypes

// host-side decode (priors.bin bundled as an asset):
//   box = SSD-decode(loc, priors, variances=[0.1,0.2]); per-class NMS (score 0.3, IoU 0.5);
//   per kept det: mask = sigmoid(proto @ coeff) (>0) cropped to the box.
// Full implementation: YolactSegmenter.kt in the sample app.

Python (LiteRT / ai-edge-litert)

import numpy as np
from ai_edge_litert.interpreter import Interpreter

it = Interpreter(model_path="yolact.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()
it.set_tensor(inp[0]["index"], x)          # [1,3,550,550] BGR, normalized (see above)
it.invoke()
outs = {tuple(o["shape"][1:]): it.get_tensor(o["index"])[0] for o in out}
loc  = outs[(19248, 4)]; conf = outs[(19248, 81)]
mask = outs[(19248, 32)]; proto = outs[(138, 138, 32)]

priors = np.fromfile("priors.bin", np.float32).reshape(-1, 4)
cxy = priors[:, :2] + loc[:, :2] * 0.1 * priors[:, 2:]
wh  = priors[:, 2:] * np.exp(loc[:, 2:] * 0.2)
boxes = np.concatenate([cxy - wh / 2, cxy + wh / 2], 1)      # x1y1x2y2 (0..1)
# then per-class NMS on conf, and mask_i = sigmoid(proto @ mask[i]) cropped to boxes[i]

License

MIT (YOLACT / dbolya/yolact). COCO class taxonomy.