README.md

---
title: Flowframes
emoji: 🎞️
colorFrom: pink
colorTo: green
sdk: gradio
sdk_version: 6.26.0
python_version: "3.10"
app_file: app.py
pinned: false
fullWidth: true
license: mit
short_description: RIFE frame interpolation, streams a clip of any length
startup_duration_timeout: 1h
---

# Flowframes

RIFE frame interpolation as a Space. Upload a clip, get it back at two or four
times the frame rate.

[Flowframes](https://github.com/n00mkrad/flowframes) does this job on Windows.
That project is a C# GUI wrapping ffmpeg and a set of interpolation backends, so
none of it ports directly. What I reimplemented is the orchestration, in one
`app.py` over one ONNX model.

## Length is free, pixels are not

Nothing accumulates in memory. ffmpeg decodes raw frames into a pipe, two frames
are held at a time, each pair is interpolated, and the results stream straight
into a second ffmpeg. A 4K frame is about 100 MB as float32, so a five minute 4K
source costs the same memory as a five second one. Only time scales.

Time scales with pixels rather than seconds, which is why "Max height" is the
control that matters most. Halving the height is roughly four times faster.

## Segments

Work is cut into segments sized from a per-frame rate measured on the hardware
that is actually running, never from a table. On ZeroGPU each segment is its own
GPU call, so the 120 s limit bounds a segment rather than the whole job and clip
length stops being a constraint. On CPU the same seams act as checkpoints, so
pressing Stop keeps everything already rendered.

The Interpolate button carries the estimate and updates as you change the clip,
the multiplier or Max height. It counts what you actually wait for, which is not
the same as inference time: the per-segment overhead the plan under-charges, and
on ZeroGPU the allocation queue, which for a short clip is the largest term of
the three. A job needing more than six hours is refused up front with a suggested
resolution, rather than discovering the problem hours in.

If a segment overruns its plan badly, the remaining segments are re-cut from what
that segment actually cost. A segment reports progress only when it finishes, so
an oversized one freezes the log and adds its whole duration to how long Stop
takes to respond.

## What it costs, measured

Every number here came off this Space.

| clip | where | result |
| --- | --- | --- |
| 6 s, 1280x720, 2x | ZeroGPU | 180 -> 359 frames, 15 s end to end |
| 2 min, 426x240, 2x | CPU, 12 threads | 3600 -> 7199 frames, 589 s, 47 segments |

Per frame at 720p: 0.021 s on the GPU, 0.83 s on a Space CPU. The header quotes
that number, and it is why the header says seconds on GPU hardware and minutes on
CPU.

That 0.021 was 0.130 until the model was converted from opset 19 to opset 18.
Nothing else changed and the two graphs are bit identical, but at opset 19 the
CUDA provider declined all 22 `Resize` nodes and ran them on the CPU, which
forced a host round trip at each one. Profiling inside the GPU call showed it:

```
opset 19  MemcpyFromHost 34%, Resize 27%, Conv 19%   memcpy 44%, on CPU EP 28%
opset 18  Conv 38%, Add 16%, LeakyRelu 8%            memcpy  0%, on CPU EP  0%
```

The copies were never input transfer. They were one op bouncing the graph between
devices twenty two times per inference, and the fix was a version number.

Frame counts are exact. A 47-segment run reassembles to 7199 frames, which is
`3600 * 2 - 1` to the frame. Segment bounds carry a one-frame overlap and only
the last segment writes its tail, so no seam duplicates or drops a frame. At 4x
the same holds across ten segments, `(180 - 1) * 4 + 1 = 717`. Audio carries over
from the source and survives the segment concat. Odd source dimensions round down
to even, because yuv420p subsamples chroma 2x2 and x264 refuses an odd side.

## Execution providers

One model file, best available provider, in order.

| provider | status |
| --- | --- |
| TensorRT | Registered, never selected, and deliberately not pursued. See below. |
| CUDA, fp32 | what runs on GPU |
| CPU, fp32 | no GPU |

The TensorRT EP ships inside the ONNX Runtime wheel, so ORT always reports it as
available, but it needs `libnvinfer.so.10`, TensorRT's own runtime, which is not
in the image. Adding it is one pinned dependency. The reason I have not: a
TensorRT engine compiles on first inference, which on ZeroGPU means inside the
120 s window, and the engine cache lives on a filesystem that dies with the
container. Every cold start would recompile and might time out. That trades a
working 0.021 s/frame for a maybe-faster path that intermittently fails to start.
The code probes for the library before attempting the provider, so a missing
runtime costs nothing rather than a failed session construction on every GPU
call, and the path lights up by itself if the image ever ships it.

Providers are tried one at a time rather than as a single list, because ORT's
list is all or nothing: a missing TensorRT takes CUDA down with it and lands on
CPU. ORT also does not raise when a provider fails. It prints `EP Error`, falls
back to CPU inside the constructor and hands back a valid session, so the code
reads the selected provider back from the session and prints it in the header. A
Space running fifty times slower than it should looks exactly like one that is
not.

fp16 is not used on the GPU either, and that was a measurement rather than an
oversight. I converted the model to fp16 and A/B'd it against fp32 on the CUDA
provider inside a single GPU allocation on one container: fp16 0.1096 s/frame
against fp32 0.0306. Three and a half times slower. Cosine agreement was 0.99994,
so a quality gate would have passed it straight through.

On the CPU path fp16 is wrong for a different reason. Those cores have no half
precision ALU, so every operation promotes to fp32 and you pay the conversion on
top.

Thread count comes from the cgroup, not `os.cpu_count()`, which reports the host.
The ZeroGPU container's `cpu.max` reads `1600000 100000`, so sixteen CPUs, and
the cap takes twelve of them: measured at 720p this graph stops scaling around
twelve threads and thirty two run 4.3x slower. Sixteen is worth knowing because
it is not the two vCPUs a free CPU Space gets, and sizing anything against two
here would be wrong by a factor of six.

## Model

RIFE 4.9, ONNX, 21 MB, committed to this repo rather than downloaded at startup,
so a wake costs nothing and no upstream repo can disappear from under it.

Stored at opset 18, converted from the opset 19 export. The conversion is bit
identical on this graph, verified across three resolutions and three timesteps,
and it is what keeps `Resize` on the CUDA provider. Re-exporting at opset 19
would silently cost a factor of five.

## Credit

[RIFE](https://github.com/hzwer/Practical-RIFE) by Huang et al. ONNX export from
[yuvraj108c/rife-onnx](https://huggingface.co/yuvraj108c/rife-onnx).
Orchestration modelled on [Flowframes](https://github.com/n00mkrad/flowframes) by
n00mkrad.