LayeredFlow / README.md
Hermera's picture
Migrate LayeredFlow checkpoints from the original Drive release
b0da4ff verified
|
Raw History Blame Contribute Delete
4.04 kB
---
tags:
- optical-flow
- multi-layer-optical-flow
- pytorch
- raft
- multiraft
- layeredflow
datasets:
- Hermera/LayeredFlow-Syn
---
# LayeredFlow pre-trained checkpoints
Pre-trained RAFT and MultiRAFT checkpoints for [LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow](https://arxiv.org/abs/2409.05688) (ECCV 2024), by Hongyu Wen, Erich Liang, and Jia Deng.
These files are migrated from the checkpoint folder linked by the [official LayeredFlow repository](https://github.com/princeton-vl/LayeredFlow): [original Google Drive release](https://drive.google.com/drive/folders/1vB8wfeS1JpfTz-jKdiD_e_IM5Sn5Wztl). The original checkpoint filenames are preserved.
## Models
- **RAFT:** single-layer optical-flow models evaluated on the first-layer and last-layer benchmark subsets.
- **MultiRAFT:** the repository's multi-layer RAFT model, which predicts four optical-flow layers and is evaluated on the full benchmark.
The checkpoints use the original PyTorch `.pth` format and are loaded by the LayeredFlow evaluation scripts.
## Files
| Checkpoint | Use | Size (bytes) |
| --- | --- | ---: |
| `raft-sintel.pth` | RAFT Sintel checkpoint; also used for MultiRAFT initialization | 21,108,000 |
| `last_layer_L.pth` | RAFT last-layer evaluation | 21,103,087 |
| `first_layer_S+L.pth` | RAFT first-layer evaluation | 21,103,087 |
| `first_layer_S.pth` | RAFT first-layer evaluation | 21,103,087 |
| `first_layer_L.pth` | RAFT first-layer evaluation | 21,103,087 |
| `MultiRAFT_S+L.pth` | MultiRAFT multi-layer evaluation | 34,091,437 |
| `MultiRAFT_L.pth` | MultiRAFT multi-layer evaluation | 34,091,437 |
The seven checkpoints are unchanged copies of the original release. `checkpoint-manifest.json` records each source URL, byte size, and SHA-256 digest; `SHA256SUMS` contains the checksums.
## Download and evaluate
Clone the [LayeredFlow source repository](https://github.com/princeton-vl/LayeredFlow) and follow its installation and benchmark-data setup instructions. From the source repository root, download the checkpoints with the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli):
```bash
hf download Hermera/LayeredFlow --include 'first_layer_*.pth' 'last_layer_*.pth' 'raft-sintel.pth' --local-dir RAFT/checkpoints
hf download Hermera/LayeredFlow --include 'MultiRAFT_*.pth' 'raft-sintel.pth' --local-dir MultiRAFT/checkpoints
```
This places the files under `RAFT/checkpoints/` and `MultiRAFT/checkpoints/`, where the evaluation scripts expect them.
Run the following validation examples from the source repository root:
```bash
# First-layer optical flow
(cd RAFT && python3 evaluate_firstlayer.py --checkpoint first_layer_S+L.pth --dataset layeredflow)
# Last-layer optical flow
(cd RAFT && python3 evaluate_lastlayer.py --checkpoint last_layer_L.pth --dataset layeredflow)
# Multi-layer optical flow
(cd MultiRAFT && python3 evaluate.py --checkpoint MultiRAFT_S+L.pth --dataset layeredflow --mixed_precision)
```
The repository's evaluation scripts require a CUDA-capable environment. See the [benchmark website](https://layeredflow.cs.princeton.edu/) for test-set submission instructions.
## Related resources
- [Source code and benchmark documentation](https://github.com/princeton-vl/LayeredFlow)
- [LayeredFlow benchmark](https://layeredflow.cs.princeton.edu/)
- [LayeredFlow-Syn synthetic images and annotations](https://huggingface.co/datasets/Hermera/LayeredFlow-Syn)
## Attribution and license
The original LayeredFlow code is distributed under the [BSD 3-Clause license](https://github.com/princeton-vl/LayeredFlow/blob/main/LICENSE). This checkpoint migration does not introduce new license terms. Refer to the original release and its authors for checkpoint licensing.
## Citation
```bibtex
@article{wen2024layeredflow,
title={LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow},
author={Hongyu Wen and Erich Liang and Jia Deng},
journal={arXiv preprint arXiv:2409.05688},
year={2024},
}
```