litert-community/UniSal-Saliency-LiteRT

UniSal — LiteRT (on-device visual saliency prediction, fully-GPU)

UniSal (rdroste), visual saliency prediction — a heatmap of where humans look in an image — converted to LiteRT and running fully on the CompiledModel GPU (ML Drift) on Android. MobileNetV2 encoder + bilinear decoder, 3.71 M params / 6.5 MB fp16.

On-device (Pixel 8a, Tensor G3 — verified)

nodes on GPU	158 / 158 LITERT_CL (full residency)
inference	~3 ms (256×256)
size	6.5 MB (fp16)
accuracy	device-vs-PyTorch corr 0.9998

image[1,3,256,256] (ImageNet mean/std) →[GPU: UniSal]→ saliency[1,1,256,256] (higher = more attended)

Minimal usage

Android (Kotlin, CompiledModel GPU)

val model = CompiledModel.create(context.assets, "unisal_fp16.tflite",
    CompiledModel.Options(Accelerator.GPU), null)
val inputs = model.createInputBuffers()
val outputs = model.createOutputBuffers()
inputs[0].writeFloat(chw)            // [1,3,256,256] ImageNet-normalized, NCHW
model.run(inputs, outputs)
val sal = outputs[0].readFloat()    // [1,1,256,256] saliency (higher = more attended)

Python (desktop verification)

MEAN = np.array([0.485, 0.456, 0.406], np.float32)
STD  = np.array([0.229, 0.224, 0.225], np.float32)
import numpy as np
from PIL import Image
from ai_edge_litert.interpreter import Interpreter

img = Image.open("photo.jpg").convert("RGB").resize((256, 256))
x = ((np.asarray(img, np.float32) / 255 - MEAN) / STD).transpose(2, 0, 1)[None]

it = Interpreter(model_path="unisal_fp16.tflite"); it.allocate_tensors()
it.set_tensor(it.get_input_details()[0]["index"], x); it.invoke()
s = it.get_tensor(it.get_output_details()[0]["index"])[0, 0]      # [256,256]
s = (s - s.min()) / (s.max() - s.min())
Image.fromarray((s * 255).astype(np.uint8)).save("saliency.png")

How it converts (litert-torch) — three numerically-exact fixes

1. Strided subsample x[..., ::2, ::2] → F.avg_pool2d(x, 1, 2) (same pixels; avoids GATHER_ND).
2. Bake the 16 Gaussian prior maps (size-only constants; avoids GATHER_ND/BROADCAST_TO).
3. F.pad(replicate) → 0-pad for the 41×41 Gaussian smoothing (which is kept — it suppresses border artifacts, not cosmetic).

Result: banned ops NONE, ≤4D, tflite-vs-torch corr 1.0, device-vs-torch corr 0.9998. Static-image path (Bypass-RNN + SALICON domain pinned); the spatial log-softmax / normalization runs in the app.

Preprocessing

Center-crop, resize 256×256, /255, ImageNet mean/std, NCHW.

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
LiteRT CompiledModel (LITERT_CL)	GPU	158 / 158	~3 ms
TFLite benchmark_model (TfLiteGpuDelegateV2)	GPU (OpenCL)	158 / 158	20.2 ms
TFLite benchmark_model	CPU (XNNPACK, 4 threads)	—	XNNPACK declined the graph

The two GPU rows are different runtimes, not a contradiction. The LITERT_CL figure is the one recorded when this model shipped, taken through LiteRT's own CompiledModel accelerator — the path the Kotlin sample app and the LiteRT API use. The TfLiteGpuDelegateV2 figure is the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. They agree on how much of the graph the GPU takes; they disagree on speed, and the classic delegate is the slower of the two here. Read the TfLiteGpuDelegateV2 row as a reproducible floor, not as this model's speed on LiteRT.

XNNPACK declines these fp16 graphs — it reports failed to delegate DEPTHWISE_CONV_2D and then fails to allocate tensors — so there is no usable CPU number. Disabling XNNPACK falls back to reference kernels, which measured about 20× slower than the GPU on models of this size and would not represent CPU inference anyone would ship.

License

Apache-2.0. Upstream: rdroste/unisal.