File size: 4,882 Bytes
ebec118 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 | ---
license: mit
library_name: coreml
tags:
- frame-interpolation
- video-frame-interpolation
- coreml
- ios
- on-device
- rife
pipeline_tag: image-to-image
---
# LiquidInbetween
A small learned in-betweener for photos that runs on the phone. Give it two
photos and a moment between them, and it returns the frame at that moment.
People are part-way into their next pose and things are part-way to where
they went. Parts that appear or disappear are handled by the network, so
nothing is smeared or double-exposed.
  
## Use it in an app (Core ML)
Add `LiquidInbetweenNet.mlpackage` to your Xcode target. The inputs are
`first` and `second` (512 × 512 images) and `t` (a 1 × 1 × 1 × 1 float); the
output `frame` is the 512 × 512 frame at that moment.
```swift
import CoreML
import Vision
let model = try LiquidInbetweenNet(configuration: MLModelConfiguration()).model
let fill: [MLFeatureValue.ImageOption: Any] = [.cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue]
let first = try MLFeatureValue(cgImage: photoA, pixelsWide: 512, pixelsHigh: 512,
pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let second = try MLFeatureValue(cgImage: photoB, pixelsWide: 512, pixelsHigh: 512,
pixelFormatType: kCVPixelFormatType_32BGRA, options: fill)
let t = try MLMultiArray(shape: [1, 1, 1, 1], dataType: .float32)
t[0] = 0.5
let output = try model.prediction(from: MLDictionaryFeatureProvider(dictionary: [
"first": first, "second": second, "t": MLFeatureValue(multiArray: t),
]))
let frame = output.featureValue(for: "frame")?.imageBufferValue // 512 × 512
```
Both photos are stretched to fill the square, so the motion between them
stays consistent; stretch each frame back to your photos' shape to show it.
## Use it in Python
```python
import torch
from safetensors.torch import load_file
from liquid_model import IFNet
net = IFNet(timestep=True).eval()
net.load_state_dict(load_file("liquid_inbetween.safetensors"))
# first, second: 1 × 3 × H × W tensors in 0…1, sides divisible by 32.
with torch.no_grad():
frame = net(first, second, t=torch.tensor([[[[0.5]]]]))
```
## The model
- **Architecture.** The model is RIFE's IFNet (Huang et al., *Real-Time
Intermediate Flow Estimation for Video Frame Interpolation*, ECCV 2022,
MIT). It predicts flows from the in-between moment back to both photos,
plus an occlusion mask that decides which photo each pixel trusts. It has
about 2.3 M parameters.
- **What's new: any moment in one pass.** Every block also sees the timestep
t, and the mirrored pass sees 1 − t. RIFE alone only makes the exact
middle and needs recursion for other moments. The new input channel starts
at zero, so training begins exactly at RIFE's behaviour.
- **Fine-tuned for photos, not video.** Each training pair is two frames
0.13–0.8 s apart (photos taken moments apart), much larger gaps than
consecutive video frames. The target is a real frame somewhere between
them, at its own t. The footage is public-domain and CC0 Wikimedia
Commons video of people moving: dance, children playing, NASA astronaut
reels and military PT. See `training/SOURCES.tsv`.
- **Core ML.** The weights are FP16, about 8.6 MB, with image inputs and
output. The conversion was measured at 42.9 dB PSNR against the PyTorch
model. 8-bit weights halved the size but cost too much accuracy, so they
are not used.
Training, evaluation and export scripts are in `training/`.
## Files
- `LiquidInbetweenNet.mlpackage`: Core ML, FP16. Inputs are `first` and
`second` (512 × 512 RGB images) and `t` (1×1×1×1 float); the output is
`frame`.
- `liquid_inbetween.safetensors`: PyTorch weights for `IFNet(timestep=True)`
in `liquid_model.py`.
- `review.mp4`: the model's frames beside a plain cross-fade, on held-out
footage.
## Results on held-out footage
The evaluation set is a different video from any used in training: a
public-domain folk dance, with gaps of 0.4–0.8 s. Scores are PSNR (dB) at
t = ¼, ½ and ¾.
| | Whole frame | Moving pixels only |
|---|---|---|
| Plain cross-fade | 36.97 | 22.74 |
| RIFE (recursive for ¼, ¾) | 36.50 | 22.60 |
| **LiquidInbetween** (one pass) | **36.77** | **22.92** |
PSNR rewards blurry cross-fades, so it undersells sharp in-betweens: compare
`review.mp4`.
## Limits
This is an in-betweener, not a generative video model. It moves what is in
the two photos and does not invent content that is in neither. Two
unrelated photos become a flowing dissolve, not a semantic morph.
## License
MIT, as is RIFE's code and weights it starts from (© 2021 hzwer). Made by
[Haplo LLC](https://haploapp.com).
|