README.md

---
license: apache-2.0
library_name: litert
pipeline_tag: video-classification
tags:
  - litert
  - tflite
  - android
  - on-device
  - gpu
  - video-action-recognition
  - kinetics-600
  - movinet
  - streaming
---

# MoViNet-A0 Stream — LiteRT (on-device video action recognition, GPU)

On-device **streaming video action recognition**: recognises human actions across a
stream of camera frames — one frame at a time, constant memory, real-time — running
**fully on the LiteRT `CompiledModel` GPU delegate** (no CPU fallback).

- **Architecture:** [MoViNet-A0](https://arxiv.org/abs/2103.11511) streaming variant
  (Google Research) — a causal 2+1D CNN.
- **Task:** [Kinetics-600](https://github.com/cvdfoundation/kinetics-dataset) — 600 action classes.
- **Weights:** ported PyTorch checkpoint from [Atze00/MoViNet-pytorch](https://github.com/Atze00/MoViNet-pytorch).
- **Size:** 15 MB · ~3.75 M params · input frame `172×172`.

![MoViNet-A0 streaming action recognition](hero.png)

## How the streaming graph works

MoViNet's temporal convolutions and global-average-pools each keep a small buffer of
the recent past, so the network can be fed **one frame at a time** and its prediction
sharpens as more frames of the same action arrive. The stock streaming graph carries
that history in **5D** state tensors `[1, T, H, W, C]`, which a GPU delegate cannot
compile (all tensors must be ≤ 4D). This model is re-authored as a **single-frame,
4D-only functional forward** (**47 inputs / 28 outputs**) with the recurrent state
threaded explicitly through the graph I/O:

| I/O slot        | count | shape           | meaning                                   |
|-----------------|-------|-----------------|-------------------------------------------|
| `input[0]`      | 1     | `[1,3,172,172]` | current RGB frame (NCHW, 0..1)            |
| `input[1..28]`  | 28    | `[1,C,H,W]`     | temporal-conv stream buffers (11 convs)   |
| `input[29..44]` | 16    | `[1,C,1,1]`     | streaming avg-pool running sums (15 SE + head) |
| `input[45]`     | 1     | `[1,1,1,1]`     | `inv_count` = 1 / current frame number    |
| `input[46]`     | 1     | `[1,1,1,1]`     | constant `1.0` (Mali output decoupler)    |
| `output[0]`     | 1     | `[1,600]`       | Kinetics-600 logits                       |
| `output[1..11]` | 11    | `[1,C,H,W]`     | current per-temporal-conv frame           |
| `output[12..27]`| 16    | `[1,C,1,1]`     | fresh per-frame spatial means             |

The **stream-buffer shift register and pool running-sum accumulation are done
host-side**: each frame you run once, shift each stream buffer (drop oldest, append
the emitted current frame), accumulate `running_sum += emitted_mean`, and feed both
back as inputs. The converted graph is **all float32, 0 tensors of rank > 4, 0
GPU-incompatible ops** and matches the original PyTorch model bit-for-bit
(correlation 0.99999999999, top-5 identical; device GPU on a Pixel 8a locks onto
"jumping jacks" within a few frames). Keeping the state in-graph tripped three silent
Mali `CompiledModel` bugs, which is why the state plumbing is host-side.

## Minimal usage

### Python (LiteRT / ai-edge-litert, frame-by-frame)

```python
from ai_edge_litert.interpreter import Interpreter
import numpy as np

it = Interpreter(model_path="movinet_a0_stream.tflite"); it.allocate_tensors()
inp, out = it.get_input_details(), it.get_output_details()

DIMS = [2, 2, 2, 4, 2, 2, 4, 2, 2, 2, 4]        # temporal-conv buffer depths
offs, o = [], 0
for d in DIMS: offs.append(o); o += d
hist = [[np.zeros(inp[1 + offs[c] + i]["shape"], np.float32) for i in range(DIMS[c])]
        for c in range(11)]                      # host-side shift registers
psum = [np.zeros(inp[29 + i]["shape"], np.float32) for i in range(16)]  # running sums

for n, frame in enumerate(video_frames, start=1):   # frame: [1,3,172,172], RGB, 0..1
    it.set_tensor(inp[0]["index"], frame.astype(np.float32))
    for c in range(11):
        for i in range(DIMS[c]): it.set_tensor(inp[1 + offs[c] + i]["index"], hist[c][i])
    for i in range(16): it.set_tensor(inp[29 + i]["index"], psum[i])
    it.set_tensor(inp[45]["index"], np.full((1, 1, 1, 1), 1.0 / n, np.float32))  # inv_count
    it.set_tensor(inp[46]["index"], np.ones((1, 1, 1, 1), np.float32))           # decoupler
    it.invoke()
    logits = it.get_tensor(out[0]["index"])[0]      # [600]
    for c in range(11):                             # shift: drop oldest, append current
        hist[c] = hist[c][1:] + [it.get_tensor(out[1 + c]["index"]).copy()]
    for i in range(16):                             # accumulate running sum
        psum[i] = psum[i] + it.get_tensor(out[12 + i]["index"])

print("top-1:", int(logits.argmax()))
```

### Kotlin (Android, LiteRT CompiledModel GPU)

```kotlin
val options = CompiledModel.Options(Accelerator.GPU)
val model = CompiledModel.create(context.assets, "movinet_a0_stream.tflite", options, null)
val inBufs = model.createInputBuffers()    // [0]=frame, [1..28]=stream, [29..44]=pool sums, [45]=inv_count, [46]=1.0
val outBufs = model.createOutputBuffers()  // [0]=logits, [1..11]=current frames, [12..27]=fresh means

inBufs[46].writeFloat(floatArrayOf(1f))    // constant decoupler
// reset recurrent state (zeros) at the start of a clip; keep host-side stream buffers + pool sums

for ((n, frameNCHW) in videoFrames.withIndex()) {           // frame: [1,3,172,172], RGB, 0..1
    inBufs[0].writeFloat(frameNCHW)
    inBufs[45].writeFloat(floatArrayOf(1f / (n + 1)))       // inv_count
    // (stream inputs 1..28 and pool inputs 29..44 already staged from the previous frame)
    model.run(inBufs, outBufs)
    val logits = outBufs[0].readFloat()                     // [600] Kinetics-600
    // host-side: shift each stream buffer with the emitted current frame (outBufs[1..11]),
    // and accumulate poolSum[i] += outBufs[12+i], then write both back to inBufs for the next frame.
}
```

A full implementation (camera → per-frame → top-5, with the host-side shift register and pool
accumulation) is in the sample app's `ActionRecognizer.kt`.

## Conversion

Re-authored and converted with **litert-torch**. See the sample app and build script:
`build_movinet.py` + `stream_model.py`.

## License

Apache-2.0 (MoViNet / Atze00/MoViNet-pytorch). Kinetics-600 label taxonomy from the
DeepMind Kinetics dataset.