LiquidInbetween

A small learned in-betweener for photos that runs on the phone. Give it two photos and a moment between them, and it returns the frame at that moment. People are part-way into their next pose and things are part-way to where they went. Parts that appear or disappear are handled by the network, so nothing is smeared or double-exposed.

Use it in an app (Core ML)

Add LiquidInbetweenNet.mlpackage to your Xcode target. The inputs are first and second (512 × 512 images) and t (a 1 × 1 × 1 × 1 float); the output frame is the 512 × 512 frame at that moment.

import CoreML
import Vision

let model = try LiquidInbetweenNet(configuration: MLModelConfiguration()).model
let fill: [MLFeatureValue.ImageOption: Any] = [.cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue]
let first = try MLFeatureValue(cgImage: photoA, pixelsWide: 512, pixelsHigh: 512,
                               pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let second = try MLFeatureValue(cgImage: photoB, pixelsWide: 512, pixelsHigh: 512,
                                pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let t = try MLMultiArray(shape: [1, 1, 1, 1], dataType: .float32)
t[0] = 0.5
let output = try model.prediction(from: MLDictionaryFeatureProvider(dictionary: [
    "first": first, "second": second, "t": MLFeatureValue(multiArray: t),
]))
let frame = output.featureValue(for: "frame")?.imageBufferValue   // 512 × 512

Both photos are stretched to fill the square, so the motion between them stays consistent; stretch each frame back to your photos' shape to show it.

Use it in Python

import torch
from safetensors.torch import load_file
from liquid_model import IFNet

net = IFNet(timestep=True).eval()
net.load_state_dict(load_file("liquid_inbetween.safetensors"))
# first, second: 1 × 3 × H × W tensors in 0…1, sides divisible by 32.
with torch.no_grad():
    frame = net(first, second, t=torch.tensor([[[[0.5]]]]))

The model

Architecture. The model is RIFE's IFNet (Huang et al., Real-Time Intermediate Flow Estimation for Video Frame Interpolation, ECCV 2022, MIT). It predicts flows from the in-between moment back to both photos, plus an occlusion mask that decides which photo each pixel trusts. It has about 2.3 M parameters.
What's new: any moment in one pass. Every block also sees the timestep t, and the mirrored pass sees 1 − t. RIFE alone only makes the exact middle and needs recursion for other moments. The new input channel starts at zero, so training begins exactly at RIFE's behaviour.
Fine-tuned for photos, not video. Each training pair is two frames 0.13–0.8 s apart (photos taken moments apart), much larger gaps than consecutive video frames. The target is a real frame somewhere between them, at its own t. The footage is public-domain and CC0 Wikimedia Commons video of people moving: dance, children playing, NASA astronaut reels and military PT. See training/SOURCES.tsv.
Core ML. The weights are FP16, about 8.6 MB, with image inputs and output. The conversion was measured at 42.9 dB PSNR against the PyTorch model. 8-bit weights halved the size but cost too much accuracy, so they are not used.

Training, evaluation and export scripts are in training/.

Files

LiquidInbetweenNet.mlpackage: Core ML, FP16. Inputs are first and second (512 × 512 RGB images) and t (1×1×1×1 float); the output is frame.
liquid_inbetween.safetensors: PyTorch weights for IFNet(timestep=True) in liquid_model.py.
review.mp4: the model's frames beside a plain cross-fade, on held-out footage.

Results on held-out footage

The evaluation set is a different video from any used in training: a public-domain folk dance, with gaps of 0.4–0.8 s. Scores are PSNR (dB) at t = ¼, ½ and ¾.

	Whole frame	Moving pixels only
Plain cross-fade	36.97	22.74
RIFE (recursive for ¼, ¾)	36.50	22.60
LiquidInbetween (one pass)	36.77	22.92

PSNR rewards blurry cross-fades, so it undersells sharp in-betweens: compare review.mp4.

Limits

This is an in-betweener, not a generative video model. It moves what is in the two photos and does not invent content that is in neither. Two unrelated photos become a flowing dissolve, not a semantic morph.

License

Downloads last month: 6