---
license: mit
library_name: coreml
tags:
- frame-interpolation
- video-frame-interpolation
- coreml
- ios
- on-device
- rife
pipeline_tag: image-to-image
---

# LiquidInbetween

A small learned in-betweener for photos that runs on the phone. Give it two
photos and a moment between them, and it returns the frame at that moment.
People are part-way into their next pose and things are part-way to where
they went. Parts that appear or disappear are handled by the network, so
nothing is smeared or double-exposed.

![iOS 17+](https://img.shields.io/badge/iOS-17%2B-blue) ![Core ML](https://img.shields.io/badge/Core%20ML-8.6%20MB-green) ![MIT](https://img.shields.io/badge/license-MIT-lightgrey)

## Use it in an app (Core ML)

Add `LiquidInbetweenNet.mlpackage` to your Xcode target. The inputs are
`first` and `second` (512 × 512 images) and `t` (a 1 × 1 × 1 × 1 float); the
output `frame` is the 512 × 512 frame at that moment.

```swift
import CoreML
import Vision

let model = try LiquidInbetweenNet(configuration: MLModelConfiguration()).model
let fill: [MLFeatureValue.ImageOption: Any] = [.cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue]
let first = try MLFeatureValue(cgImage: photoA, pixelsWide: 512, pixelsHigh: 512,
                               pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let second = try MLFeatureValue(cgImage: photoB, pixelsWide: 512, pixelsHigh: 512,
                                pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let t = try MLMultiArray(shape: [1, 1, 1, 1], dataType: .float32)
t[0] = 0.5
let output = try model.prediction(from: MLDictionaryFeatureProvider(dictionary: [
    "first": first, "second": second, "t": MLFeatureValue(multiArray: t),
]))
let frame = output.featureValue(for: "frame")?.imageBufferValue   // 512 × 512
```

Both photos are stretched to fill the square, so the motion between them
stays consistent; stretch each frame back to your photos' shape to show it.

## Use it in Python

```python
import torch
from safetensors.torch import load_file
from liquid_model import IFNet

net = IFNet(timestep=True).eval()
net.load_state_dict(load_file("liquid_inbetween.safetensors"))
# first, second: 1 × 3 × H × W tensors in 0…1, sides divisible by 32.
with torch.no_grad():
    frame = net(first, second, t=torch.tensor([[[[0.5]]]]))
```

## The model

- **Architecture.** The model is RIFE's IFNet (Huang et al., *Real-Time
  Intermediate Flow Estimation for Video Frame Interpolation*, ECCV 2022,
  MIT). It predicts flows from the in-between moment back to both photos,
  plus an occlusion mask that decides which photo each pixel trusts. It has
  about 2.3 M parameters.
- **What's new: any moment in one pass.** Every block also sees the timestep
  t, and the mirrored pass sees 1 − t. RIFE alone only makes the exact
  middle and needs recursion for other moments. The new input channel starts
  at zero, so training begins exactly at RIFE's behaviour.
- **Fine-tuned for photos, not video.** Each training pair is two frames
  0.13–0.8 s apart (photos taken moments apart), much larger gaps than
  consecutive video frames. The target is a real frame somewhere between
  them, at its own t. The footage is public-domain and CC0 Wikimedia
  Commons video of people moving: dance, children playing, NASA astronaut
  reels and military PT. See `training/SOURCES.tsv`.
- **Core ML.** The weights are FP16, about 8.6 MB, with image inputs and
  output. The conversion was measured at 42.9 dB PSNR against the PyTorch
  model. 8-bit weights halved the size but cost too much accuracy, so they
  are not used.

Training, evaluation and export scripts are in `training/`.

## Files

- `LiquidInbetweenNet.mlpackage`: Core ML, FP16. Inputs are `first` and
  `second` (512 × 512 RGB images) and `t` (1×1×1×1 float); the output is
  `frame`.
- `liquid_inbetween.safetensors`: PyTorch weights for `IFNet(timestep=True)`
  in `liquid_model.py`.
- `review.mp4`: the model's frames beside a plain cross-fade, on held-out
  footage.

## Results on held-out footage

The evaluation set is a different video from any used in training: a
public-domain folk dance, with gaps of 0.4–0.8 s. Scores are PSNR (dB) at
t = ¼, ½ and ¾.

| | Whole frame | Moving pixels only |
|---|---|---|
| Plain cross-fade | 36.97 | 22.74 |
| RIFE (recursive for ¼, ¾) | 36.50 | 22.60 |
| **LiquidInbetween** (one pass) | **36.77** | **22.92** |

PSNR rewards blurry cross-fades, so it undersells sharp in-betweens: compare
`review.mp4`.

## Limits

This is an in-betweener, not a generative video model. It moves what is in
the two photos and does not invent content that is in neither. Two
unrelated photos become a flowing dissolve, not a semantic morph.

## License

MIT, as is RIFE's code and weights it starts from (© 2021 hzwer). Made by
[Haplo LLC](https://haploapp.com).