--- tags: - optical-flow - multi-layer-optical-flow - pytorch - raft - multiraft - layeredflow datasets: - Hermera/LayeredFlow-Syn --- # LayeredFlow pre-trained checkpoints Pre-trained RAFT and MultiRAFT checkpoints for [LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow](https://arxiv.org/abs/2409.05688) (ECCV 2024), by Hongyu Wen, Erich Liang, and Jia Deng. These files are migrated from the checkpoint folder linked by the [official LayeredFlow repository](https://github.com/princeton-vl/LayeredFlow): [original Google Drive release](https://drive.google.com/drive/folders/1vB8wfeS1JpfTz-jKdiD_e_IM5Sn5Wztl). The original checkpoint filenames are preserved. ## Models - **RAFT:** single-layer optical-flow models evaluated on the first-layer and last-layer benchmark subsets. - **MultiRAFT:** the repository's multi-layer RAFT model, which predicts four optical-flow layers and is evaluated on the full benchmark. The checkpoints use the original PyTorch `.pth` format and are loaded by the LayeredFlow evaluation scripts. ## Files | Checkpoint | Use | Size (bytes) | | --- | --- | ---: | | `raft-sintel.pth` | RAFT Sintel checkpoint; also used for MultiRAFT initialization | 21,108,000 | | `last_layer_L.pth` | RAFT last-layer evaluation | 21,103,087 | | `first_layer_S+L.pth` | RAFT first-layer evaluation | 21,103,087 | | `first_layer_S.pth` | RAFT first-layer evaluation | 21,103,087 | | `first_layer_L.pth` | RAFT first-layer evaluation | 21,103,087 | | `MultiRAFT_S+L.pth` | MultiRAFT multi-layer evaluation | 34,091,437 | | `MultiRAFT_L.pth` | MultiRAFT multi-layer evaluation | 34,091,437 | The seven checkpoints are unchanged copies of the original release. `checkpoint-manifest.json` records each source URL, byte size, and SHA-256 digest; `SHA256SUMS` contains the checksums. ## Download and evaluate Clone the [LayeredFlow source repository](https://github.com/princeton-vl/LayeredFlow) and follow its installation and benchmark-data setup instructions. From the source repository root, download the checkpoints with the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/guides/cli): ```bash hf download Hermera/LayeredFlow --include 'first_layer_*.pth' 'last_layer_*.pth' 'raft-sintel.pth' --local-dir RAFT/checkpoints hf download Hermera/LayeredFlow --include 'MultiRAFT_*.pth' 'raft-sintel.pth' --local-dir MultiRAFT/checkpoints ``` This places the files under `RAFT/checkpoints/` and `MultiRAFT/checkpoints/`, where the evaluation scripts expect them. Run the following validation examples from the source repository root: ```bash # First-layer optical flow (cd RAFT && python3 evaluate_firstlayer.py --checkpoint first_layer_S+L.pth --dataset layeredflow) # Last-layer optical flow (cd RAFT && python3 evaluate_lastlayer.py --checkpoint last_layer_L.pth --dataset layeredflow) # Multi-layer optical flow (cd MultiRAFT && python3 evaluate.py --checkpoint MultiRAFT_S+L.pth --dataset layeredflow --mixed_precision) ``` The repository's evaluation scripts require a CUDA-capable environment. See the [benchmark website](https://layeredflow.cs.princeton.edu/) for test-set submission instructions. ## Related resources - [Source code and benchmark documentation](https://github.com/princeton-vl/LayeredFlow) - [LayeredFlow benchmark](https://layeredflow.cs.princeton.edu/) - [LayeredFlow-Syn synthetic images and annotations](https://huggingface.co/datasets/Hermera/LayeredFlow-Syn) ## Attribution and license The original LayeredFlow code is distributed under the [BSD 3-Clause license](https://github.com/princeton-vl/LayeredFlow/blob/main/LICENSE). This checkpoint migration does not introduce new license terms. Refer to the original release and its authors for checkpoint licensing. ## Citation ```bibtex @article{wen2024layeredflow, title={LayeredFlow: A Real-World Benchmark for Non-Lambertian Multi-Layer Optical Flow}, author={Hongyu Wen and Erich Liang and Jia Deng}, journal={arXiv preprint arXiv:2409.05688}, year={2024}, } ```