File size: 2,720 Bytes
280576a 02629d2 280576a 6fbb750 02629d2 280576a 17c366a 280576a 02629d2 280576a 39f684e e8ceba1 39f684e e8ceba1 39f684e e8ceba1 430fd17 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
language:
- en
license: mit
metrics:
- mean_iou
tags:
- Semantic_Future_Prediction
- Video_Generation
pipeline_tag: video-to-video
---
# Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers (CVPR 2025)
This model is described in the paper [Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers](https://huggingface.co/papers/2501.08303).
Project Page: [https://futurist-cvpr2025.github.io](https://futurist-cvpr2025.github.io)
FUTURIST employs a multimodal visual sequence transformer to directly predict multiple future semantic modalities. We focus on two key modalities: semantic segmentation and depth estimation.
- Key innovation 1: We introduce a VAE-free hierarchical tokenization process integrated directly into our transformer. This simplifies training, reduces computational overhead, and enables true end-to-end optimization
- Key innovation 2: Our model features an efficient cross-modality fusion mechanism that improves predictions by learning synergies between different modalities (segmentation + depth)
- Key innovation 3: We developed a novel multimodal masked visual modeling objective specifically designed for future prediction tasks
We achieve state-of-the-art performance in future semantic segmentation on Cityscapes, with strong improvements in both short-term (0.18s) and mid-term (0.54s) predictions
# Code
https://github.com/Sta8is/FUTURIST
# Demo:
We provide 2 quick demos.
- [](https://colab.research.google.com/drive/1fS51KGb1nwDiLplVcM4stypQe3Qtb5iW?usp=sharing)
- [Demo](https://github.com/Sta8is/FUTURIST/blob/main/demo.ipynb).
# Citation:
If you found Futurist useful in your research, please consider starring ⭐ us on GitHub and citing 📚 us in your research!
```
@InProceedings{Karypidis_2025_CVPR,
author = {Karypidis, Efstathios and Kakogeorgiou, Ioannis and Gidaris, Spyros and Komodakis, Nikos},
title = {Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers},
booktitle = {Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR)},
month = {June},
year = {2025},
pages = {3793-3803}
@article{karypidis2025advancingsemanticfutureprediction,
title={Advancing Semantic Future Prediction through Multimodal Visual Sequence Transformers},
author={Efstathios Karypidis and Ioannis Kakogeorgiou and Spyros Gidaris and Nikos Komodakis},
year={2025},
journal={arXiv:2501.08303}
url={https://arxiv.org/abs/2501.08303},
}
``` |