litert-community/MoViNet-A0-Stream-LiteRT

MoViNet-A0 Stream — LiteRT (on-device video action recognition, GPU)

On-device streaming video action recognition: recognises human actions across a stream of camera frames — one frame at a time, constant memory, real-time — running fully on the LiteRT CompiledModel GPU delegate (no CPU fallback).

• Architecture: MoViNet-A0 streaming variant (Google Research) — a causal 2+1D CNN.
• Task: Kinetics-600 — 600 action classes.
• Weights: ported PyTorch checkpoint from Atze00/MoViNet-pytorch.
• Size: 15 MB · ~3.75 M params · input frame 172×172.

How the streaming graph works

MoViNet's temporal convolutions and global-average-pools each keep a small buffer of the recent past, so the network can be fed one frame at a time and its prediction sharpens as more frames of the same action arrive. The stock streaming graph carries that history in 5D state tensors [1, T, H, W, C], which a GPU delegate cannot compile (all tensors must be ≤ 4D). This model is re-authored as a single-frame, 4D-only functional forward (47 inputs / 28 outputs) with the recurrent state threaded explicitly through the graph I/O:

I/O slot	count	shape	meaning
input[0]	1	[1,3,172,172]	current RGB frame (NCHW, 0..1)
input[1..28]	28	[1,C,H,W]	temporal-conv stream buffers (11 convs)
input[29..44]	16	[1,C,1,1]	streaming avg-pool running sums (15 SE + head)
input[45]	1	[1,1,1,1]	inv_count = 1 / current frame number
input[46]	1	[1,1,1,1]	constant 1.0 (Mali output decoupler)
output[0]	1	[1,600]	Kinetics-600 logits
output[1..11]	11	[1,C,H,W]	current per-temporal-conv frame
output[12..27]	16	[1,C,1,1]	fresh per-frame spatial means

The stream-buffer shift register and pool running-sum accumulation are done host-side: each frame you run once, shift each stream buffer (drop oldest, append the emitted current frame), accumulate running_sum += emitted_mean, and feed both back as inputs. The converted graph is all float32, 0 tensors of rank > 4, 0 GPU-incompatible ops and matches the original PyTorch model bit-for-bit (correlation 0.99999999999, top-5 identical; device GPU on a Pixel 8a locks onto "jumping jacks" within a few frames). Keeping the state in-graph tripped three silent Mali CompiledModel bugs, which is why the state plumbing is host-side.

Minimal usage

Python (LiteRT / ai-edge-litert, frame-by-frame)

from ai_edge_litert.interpreter import Interpreter
import numpy as np

it = Interpreter(model_path="movinet_a0_stream.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()

DIMS = [2, 2, 2, 4, 2, 2, 4, 2, 2, 2, 4]        # temporal-conv buffer depths
offs, o = [], 0
for d in DIMS: offs.append(o); o += d
hist = [[np.zeros(inp[1 + offs[c] + i]["shape"], np.float32) for i in range(DIMS[c])]
        for c in range(11)]                      # host-side shift registers
psum = [np.zeros(inp[29 + i]["shape"], np.float32) for i in range(16)]  # running sums

for n, frame in enumerate(video_frames, start=1):   # frame: [1,3,172,172], RGB, 0..1
    it.set_tensor(inp[0]["index"], frame.astype(np.float32))
    for c in range(11):
        for i in range(DIMS[c]): it.set_tensor(inp[1 + offs[c] + i]["index"], hist[c][i])
    for i in range(16): it.set_tensor(inp[29 + i]["index"], psum[i])
    it.set_tensor(inp[45]["index"], np.full((1, 1, 1, 1), 1.0 / n, np.float32))  # inv_count
    it.set_tensor(inp[46]["index"], np.ones((1, 1, 1, 1), np.float32))           # decoupler
    it.invoke()
    logits = it.get_tensor(out[0]["index"])[0]      # [600]
    for c in range(11):                             # shift: drop oldest, append current
        hist[c] = hist[c][1:] + [it.get_tensor(out[1 + c]["index"]).copy()]
    for i in range(16):                             # accumulate running sum
        psum[i] = psum[i] + it.get_tensor(out[12 + i]["index"])

print("top-1:", int(logits.argmax()))

Kotlin (Android, LiteRT CompiledModel GPU)

val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "movinet_a0_stream.tflite", options, null)
val inBufs = model.createInputBuffers()    // [0]=frame, [1..28]=stream, [29..44]=pool sums, [45]=inv_count, [46]=1.0
val outBufs = model.createOutputBuffers()  // [0]=logits, [1..11]=current frames, [12..27]=fresh means

inBufs[46].writeFloat(floatArrayOf(1f))    // constant decoupler
// reset recurrent state (zeros) at the start of a clip; keep host-side stream buffers + pool sums

for ((n, frameNCHW) in videoFrames.withIndex()) {           // frame: [1,3,172,172], RGB, 0..1
    inBufs[0].writeFloat(frameNCHW)
    inBufs[45].writeFloat(floatArrayOf(1f / (n + 1)))       // inv_count
    // (stream inputs 1..28 and pool inputs 29..44 already staged from the previous frame)
    model.run(inBufs, outBufs)
    val logits = outBufs[0].readFloat()                     // [600] Kinetics-600
    // host-side: shift each stream buffer with the emitted current frame (outBufs[1..11]),
    // and accumulate poolSum[i] += outBufs[12+i], then write both back to inBufs for the next frame.
}

A full implementation (camera → per-frame → top-5, with the host-side shift register and pool accumulation) is in the sample app's ActionRecognizer.kt.

Conversion

Re-authored and converted with litert-torch. See the sample app and build script: build_movinet.py + stream_model.py.

Performance

Measured on a Pixel 8a (Tensor G3, Android 16) with the standard TFLite benchmark_model tool — 10 warm-up runs then 50 timed runs, reported as the tool's mean.

Runtime	Backend	Graph on GPU	Latency
TFLite benchmark_model (TfLiteGpuDelegateV2)	GPU (OpenCL)	71 / 455	53.0 ms
TFLite benchmark_model	CPU (XNNPACK, 4 threads)	—	9.8 ms

Any on-device figure recorded when this model shipped came from a different runtime. It was taken through LiteRT's own CompiledModel accelerator (logcat reports it as LITERT_CL), which is the path the Kotlin sample app and the LiteRT API use, and it appears elsewhere on this card. The rows above are the classic TFLite OpenCL delegate, measured with a tool anyone can download and re-run. The two are not comparable, so read the rows above as a reproducible floor rather than as this model's speed on LiteRT.

On this delegate the CPU is the faster choice for (9.8 ms on CPU against 53.0 ms on GPU) — worth knowing before you reach for the GPU on a mid-range phone.

Note that the GPU does not take the whole graph here (71 / 455); the remainder runs on the CPU and the split costs a per-partition round trip.

Snapdragon NPU (Hexagon)

The GPU delegate declines this graph on the S26: LiteRtException: Failed to compile model. The NPU runs it at 2.15 ms.

backend	inference (median / min)	load
NPU (Hexagon v81)	2.15 ms / 2.09 ms	112 ms

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16), LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. The run held thermal status NONE throughout. Headroom 0.68, where 1.0 is the throttling threshold.

The NPU rows here ran artifacts compiled ahead of time for SM8850 with QAIRT 2.47.0. LiteRT can also compile for the NPU on the device at first load, which is what lets you ship the published file unchanged — that path and the ten runtime libraries it needs are in the NPU recipe, and we did not measure it here. GPU wiring is in the GPU recipe.

License

Apache-2.0 (MoViNet / Atze00/MoViNet-pytorch). Kinetics-600 label taxonomy from the DeepMind Kinetics dataset.