| --- |
| license: mit |
| library_name: coreml |
| tags: |
| - frame-interpolation |
| - video-frame-interpolation |
| - coreml |
| - ios |
| - on-device |
| - rife |
| pipeline_tag: image-to-image |
| --- |
| |
| # LiquidInbetween |
|
|
| A small learned in-betweener for photos that runs on the phone. Give it two |
| photos and a moment between them, and it returns the frame at that moment. |
| People are part-way into their next pose and things are part-way to where |
| they went. Parts that appear or disappear are handled by the network, so |
| nothing is smeared or double-exposed. |
|
|
|    |
|
|
| ## Use it in an app (Core ML) |
|
|
| Add `LiquidInbetweenNet.mlpackage` to your Xcode target. The inputs are |
| `first` and `second` (512 × 512 images) and `t` (a 1 × 1 × 1 × 1 float); the |
| output `frame` is the 512 × 512 frame at that moment. |
|
|
| ```swift |
| import CoreML |
| import Vision |
| |
| let model = try LiquidInbetweenNet(configuration: MLModelConfiguration()).model |
| let fill: [MLFeatureValue.ImageOption: Any] = [.cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue] |
| let first = try MLFeatureValue(cgImage: photoA, pixelsWide: 512, pixelsHigh: 512, |
| pixelFormatType: kCVPixelFormatType_32BGRA, options: fill) |
| let second = try MLFeatureValue(cgImage: photoB, pixelsWide: 512, pixelsHigh: 512, |
| pixelFormatType: kCVPixelFormatType_32BGRA, options: fill) |
| let t = try MLMultiArray(shape: [1, 1, 1, 1], dataType: .float32) |
| t[0] = 0.5 |
| let output = try model.prediction(from: MLDictionaryFeatureProvider(dictionary: [ |
| "first": first, "second": second, "t": MLFeatureValue(multiArray: t), |
| ])) |
| let frame = output.featureValue(for: "frame")?.imageBufferValue // 512 × 512 |
| ``` |
|
|
| Both photos are stretched to fill the square, so the motion between them |
| stays consistent; stretch each frame back to your photos' shape to show it. |
|
|
| ## Use it in Python |
|
|
| ```python |
| import torch |
| from safetensors.torch import load_file |
| from liquid_model import IFNet |
| |
| net = IFNet(timestep=True).eval() |
| net.load_state_dict(load_file("liquid_inbetween.safetensors")) |
| # first, second: 1 × 3 × H × W tensors in 0…1, sides divisible by 32. |
| with torch.no_grad(): |
| frame = net(first, second, t=torch.tensor([[[[0.5]]]])) |
| ``` |
|
|
| ## The model |
|
|
| - **Architecture.** The model is RIFE's IFNet (Huang et al., *Real-Time |
| Intermediate Flow Estimation for Video Frame Interpolation*, ECCV 2022, |
| MIT). It predicts flows from the in-between moment back to both photos, |
| plus an occlusion mask that decides which photo each pixel trusts. It has |
| about 2.3 M parameters. |
| - **What's new: any moment in one pass.** Every block also sees the timestep |
| t, and the mirrored pass sees 1 − t. RIFE alone only makes the exact |
| middle and needs recursion for other moments. The new input channel starts |
| at zero, so training begins exactly at RIFE's behaviour. |
| - **Fine-tuned for photos, not video.** Each training pair is two frames |
| 0.13–0.8 s apart (photos taken moments apart), much larger gaps than |
| consecutive video frames. The target is a real frame somewhere between |
| them, at its own t. The footage is public-domain and CC0 Wikimedia |
| Commons video of people moving: dance, children playing, NASA astronaut |
| reels and military PT. See `training/SOURCES.tsv`. |
| - **Core ML.** The weights are FP16, about 8.6 MB, with image inputs and |
| output. The conversion was measured at 42.9 dB PSNR against the PyTorch |
| model. 8-bit weights halved the size but cost too much accuracy, so they |
| are not used. |
|
|
| Training, evaluation and export scripts are in `training/`. |
|
|
| ## Files |
|
|
| - `LiquidInbetweenNet.mlpackage`: Core ML, FP16. Inputs are `first` and |
| `second` (512 × 512 RGB images) and `t` (1×1×1×1 float); the output is |
| `frame`. |
| - `liquid_inbetween.safetensors`: PyTorch weights for `IFNet(timestep=True)` |
| in `liquid_model.py`. |
| - `review.mp4`: the model's frames beside a plain cross-fade, on held-out |
| footage. |
|
|
| ## Results on held-out footage |
|
|
| The evaluation set is a different video from any used in training: a |
| public-domain folk dance, with gaps of 0.4–0.8 s. Scores are PSNR (dB) at |
| t = ¼, ½ and ¾. |
|
|
| | | Whole frame | Moving pixels only | |
| |---|---|---| |
| | Plain cross-fade | 36.97 | 22.74 | |
| | RIFE (recursive for ¼, ¾) | 36.50 | 22.60 | |
| | **LiquidInbetween** (one pass) | **36.77** | **22.92** | |
|
|
| PSNR rewards blurry cross-fades, so it undersells sharp in-betweens: compare |
| `review.mp4`. |
|
|
| ## Limits |
|
|
| This is an in-betweener, not a generative video model. It moves what is in |
| the two photos and does not invent content that is in neither. Two |
| unrelated photos become a flowing dissolve, not a semantic morph. |
|
|
| ## License |
|
|
| MIT, as is RIFE's code and weights it starts from (© 2021 hzwer). Made by |
| [Haplo LLC](https://haploapp.com). |
|
|
