license: mit
library_name: litert
pipeline_tag: image-segmentation
tags:
- litert
- tflite
- android
- on-device
- gpu
- instance-segmentation
- yolact
- coco
- real-time
YOLACT-ResNet50 — LiteRT (real-time instance segmentation, GPU)
On-device real-time instance segmentation running fully on the LiteRT
CompiledModel GPU delegate (no CPU fallback). YOLACT
(ICCV 2019) predicts per-instance COCO masks. The network (ResNet50 + FPN +
protonet + heads) runs on the GPU; the lightweight decode (NMS + linear-combination
masks) runs host-side. ~41 ms/graph on a Pixel 8a.
- Architecture: YOLACT-ResNet50 (base, no deformable conv) — pure CNN.
- Weights: dbolya/yolact (
yolact_resnet50_54_800000) · MIT. - Size: 125 MB.
Files
yolact.tflite— the GPU graph (input[1,3,550,550]NCHW).priors.bin— 19248 SSD priors[cx,cy,w,h](float32) used by the host-side box decode.
I/O
- Input:
[1, 3, 550, 550]NCHW, BGR, normalized(x - [103.94,116.78,123.68]) / [57.38,57.12,58.40](no /255). - Raw outputs:
loc [1,19248,4],conf [1,19248,81](softmax, incl. background),mask [1,19248,32](coefficients),proto [1,138,138,32](prototype masks).
Host-side decode
- Boxes: SSD
decode(loc, priors, variances=[0.1,0.2]). - NMS: per-class, score-threshold ~0.3, IoU 0.5, top-k.
- Masks (lincomb): for each kept detection,
mask = sigmoid(proto @ coeff)→ crop to the box → threshold 0.5 → upscale.
GPU conversion
Base YOLACT is a pure CNN, so the graph converts fully GPU-compatible (138/138
nodes on the delegate, 1 partition; device corr 0.99999–1.0 vs PyTorch on all four
raw outputs) with one patch: the ResNet50 stem MaxPool2d(padding=1) lowers to a
-inf PADV2 (rejected by Mali), replaced by a 0-pad + unpadded maxpool (exact
post-ReLU). The scripted FPN is made traceable by disabling YOLACT's JIT
(use_jit=False). CPU-exact vs PyTorch (corr 1.0).
Minimal usage
Kotlin (Android, LiteRT CompiledModel GPU)
val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "yolact.tflite", options, null)
val inBufs = model.createInputBuffers()
val outBufs = model.createOutputBuffers() // map by size: loc=N*4, conf=N*81, mask=N*32, proto=138*138*32
inBufs[0].writeFloat(inputNCHW) // [1,3,550,550] BGR, (x-[103.94,116.78,123.68])/[57.38,57.12,58.40]
model.run(inBufs, outBufs)
val loc = outBufs[iLoc].readFloat() // [19248*4]
val conf = outBufs[iConf].readFloat() // [19248*81] (softmax)
val mask = outBufs[iMask].readFloat() // [19248*32] coefficients
val proto = outBufs[iProto].readFloat() // [138*138*32] prototypes
// host-side decode (priors.bin bundled as an asset):
// box = SSD-decode(loc, priors, variances=[0.1,0.2]); per-class NMS (score 0.3, IoU 0.5);
// per kept det: mask = sigmoid(proto @ coeff) (>0) cropped to the box.
// Full implementation: YolactSegmenter.kt in the sample app.
Python (LiteRT / ai-edge-litert)
import numpy as np
from ai_edge_litert.interpreter import Interpreter
it = Interpreter(model_path="yolact.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()
it.set_tensor(inp[0]["index"], x) # [1,3,550,550] BGR, normalized (see above)
it.invoke()
outs = {tuple(o["shape"][1:]): it.get_tensor(o["index"])[0] for o in out}
loc = outs[(19248, 4)]; conf = outs[(19248, 81)]
mask = outs[(19248, 32)]; proto = outs[(138, 138, 32)]
priors = np.fromfile("priors.bin", np.float32).reshape(-1, 4)
cxy = priors[:, :2] + loc[:, :2] * 0.1 * priors[:, 2:]
wh = priors[:, 2:] * np.exp(loc[:, 2:] * 0.2)
boxes = np.concatenate([cxy - wh / 2, cxy + wh / 2], 1) # x1y1x2y2 (0..1)
# then per-class NMS on conf, and mask_i = sigmoid(proto @ mask[i]) cropped to boxes[i]
License
MIT (YOLACT / dbolya/yolact). COCO class taxonomy.