README.md

metadata

title: Flowframes
emoji: 🎞️
colorFrom: pink
colorTo: green
sdk: gradio
sdk_version: 6.26.0
python_version: '3.10'
app_file: app.py
pinned: false
fullWidth: true
license: mit
short_description: RIFE frame interpolation, streams a clip of any length
startup_duration_timeout: 1h

Flowframes

RIFE frame interpolation as a Space. Upload a clip, get it back at two or four times the frame rate.

Flowframes does this job on Windows. That project is a C# GUI wrapping ffmpeg and a set of interpolation backends, so none of it ports directly. What I reimplemented is the orchestration, in one app.py over one ONNX model.

Length is free, pixels are not

Nothing accumulates in memory. ffmpeg decodes raw frames into a pipe, two frames are held at a time, each pair is interpolated, and the results stream straight into a second ffmpeg. A 4K frame is about 100 MB as float32, so a five minute 4K source costs the same memory as a five second one. Only time scales.

Time scales with pixels rather than seconds, which is why "Max height" is the control that matters most. Halving the height is roughly four times faster.

Segments

Work is cut into segments sized from a per-frame rate measured on the hardware that is actually running, never from a table. On ZeroGPU each segment is its own GPU call, so the 120 s limit bounds a segment rather than the whole job and clip length stops being a constraint. On CPU the same seams act as checkpoints, so pressing Stop keeps everything already rendered.

The Interpolate button carries the estimate and updates as you change the clip, the multiplier or Max height. It counts what you actually wait for, which is not the same as inference time: the per-segment overhead the plan under-charges, and on ZeroGPU the allocation queue, which for a short clip is the largest term of the three. A job needing more than six hours is refused up front with a suggested resolution, rather than discovering the problem hours in.

If a segment overruns its plan badly, the remaining segments are re-cut from what that segment actually cost. A segment reports progress only when it finishes, so an oversized one freezes the log and adds its whole duration to how long Stop takes to respond.

What it costs, measured

Every number here came off this Space.

clip	where	result
6 s, 1280x720, 2x	ZeroGPU	180 -> 359 frames, 15 s end to end
2 min, 426x240, 2x	CPU, 12 threads	3600 -> 7199 frames, 589 s, 47 segments

Per frame at 720p: 0.021 s on the GPU, 0.83 s on a Space CPU. The header quotes that number, and it is why the header says seconds on GPU hardware and minutes on CPU.

That 0.021 was 0.130 until the model was converted from opset 19 to opset 18. Nothing else changed and the two graphs are bit identical, but at opset 19 the CUDA provider declined all 22 Resize nodes and ran them on the CPU, which forced a host round trip at each one. Profiling inside the GPU call showed it:

opset 19  MemcpyFromHost 34%, Resize 27%, Conv 19%   memcpy 44%, on CPU EP 28%
opset 18  Conv 38%, Add 16%, LeakyRelu 8%            memcpy  0%, on CPU EP  0%

The copies were never input transfer. They were one op bouncing the graph between devices twenty two times per inference, and the fix was a version number.

Frame counts are exact. A 47-segment run reassembles to 7199 frames, which is 3600 * 2 - 1 to the frame. Segment bounds carry a one-frame overlap and only the last segment writes its tail, so no seam duplicates or drops a frame. At 4x the same holds across ten segments, (180 - 1) * 4 + 1 = 717. Audio carries over from the source and survives the segment concat. Odd source dimensions round down to even, because yuv420p subsamples chroma 2x2 and x264 refuses an odd side.

Execution providers

One model file, best available provider, in order.

provider	status
TensorRT	Registered, never selected, and deliberately not pursued. See below.
CUDA, fp32	what runs on GPU
CPU, fp32	no GPU

The TensorRT EP ships inside the ONNX Runtime wheel, so ORT always reports it as available, but it needs libnvinfer.so.10, TensorRT's own runtime, which is not in the image. Adding it is one pinned dependency. The reason I have not: a TensorRT engine compiles on first inference, which on ZeroGPU means inside the 120 s window, and the engine cache lives on a filesystem that dies with the container. Every cold start would recompile and might time out. That trades a working 0.021 s/frame for a maybe-faster path that intermittently fails to start. The code probes for the library before attempting the provider, so a missing runtime costs nothing rather than a failed session construction on every GPU call, and the path lights up by itself if the image ever ships it.

Providers are tried one at a time rather than as a single list, because ORT's list is all or nothing: a missing TensorRT takes CUDA down with it and lands on CPU. ORT also does not raise when a provider fails. It prints EP Error, falls back to CPU inside the constructor and hands back a valid session, so the code reads the selected provider back from the session and prints it in the header. A Space running fifty times slower than it should looks exactly like one that is not.

fp16 is not used on the GPU either, and that was a measurement rather than an oversight. I converted the model to fp16 and A/B'd it against fp32 on the CUDA provider inside a single GPU allocation on one container: fp16 0.1096 s/frame against fp32 0.0306. Three and a half times slower. Cosine agreement was 0.99994, so a quality gate would have passed it straight through.

On the CPU path fp16 is wrong for a different reason. Those cores have no half precision ALU, so every operation promotes to fp32 and you pay the conversion on top.

Thread count comes from the cgroup, not os.cpu_count(), which reports the host. The ZeroGPU container's cpu.max reads 1600000 100000, so sixteen CPUs, and the cap takes twelve of them: measured at 720p this graph stops scaling around twelve threads and thirty two run 4.3x slower. Sixteen is worth knowing because it is not the two vCPUs a free CPU Space gets, and sizing anything against two here would be wrong by a factor of six.

Model

RIFE 4.9, ONNX, 21 MB, committed to this repo rather than downloaded at startup, so a wake costs nothing and no upstream repo can disappear from under it.

Stored at opset 18, converted from the opset 19 export. The conversion is bit identical on this graph, verified across three resolutions and three timesteps, and it is what keeps Resize on the CUDA provider. Re-exporting at opset 19 would silently cost a factor of five.

Credit

RIFE by Huang et al. ONNX export from yuvraj108c/rife-onnx. Orchestration modelled on Flowframes by n00mkrad.