IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Paper • 2506.03150 • Published • 21
import torch
from diffusers import DiffusionPipeline
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("YuanzeLin/Illumicraft-checkpoints", dtype=torch.bfloat16, device_map="cuda")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]IllumiCraft Checkpoints
This repository contains the pretrained checkpoints for:
IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation
Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Ronald Clark, Ming-Hsuan Yang
🔗 Links
- 📄 Paper: https://arxiv.org/abs/2506.03150
- 🌐 Project Page: https://yuanze-lin.me/IllumiCraft_page/
- 💻 GitHub: https://github.com/yuanze-lin/IllumiCraft
- 🎥 YouTube: https://youtu.be/qAV58sADEzo
- 🤗 Dataset: https://huggingface.co/datasets/YuanzeLin/IllumiCraft
✨ Overview
IllumiCraft is a controllable video generation framework that jointly models scene geometry and illumination dynamics. By leveraging foreground appearance, HDR lighting maps, and 3D point tracks, IllumiCraft enables temporally consistent video generation with controllable motion and relighting effects.
🚀 Usage
Please refer to the official GitHub repository for installation, dataset preparation, training, and inference instructions:
https://github.com/yuanze-lin/IllumiCraft
📖 Citation
If you find IllumiCraft useful for your research, please consider citing:
@article{lin2026illumicraft,
title={IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation},
author={Lin, Yuanze and Chen, Yi-Wen and Tsai, Yi-Hsuan and Clark, Ronald and Yang, Ming-Hsuan},
journal={Advances in Neural Information Processing Systems},
volume={38},
pages={27798--27829},
year={2026}
}
- Downloads last month
- 14