nielsr HF Staff commited on
Commit
8a65eb1
·
verified ·
1 Parent(s): 8580dc8

Improve model card metadata and content

Browse files

Hi! I'm Niels, part of the community science team at Hugging Face. I've opened this PR to enhance the model card for Lumen.

Key improvements include:
- Updated the `pipeline_tag` to `image-to-video` to better reflect the video relighting/editing nature of the model.
- Added links to the paper and project page.
- Included the full list of authors and a BibTeX citation.
- Added a "Quick Start" section with environment setup and inference instructions based on your GitHub repository.

Files changed (1) hide show
  1. README.md +49 -8
README.md CHANGED
@@ -1,24 +1,65 @@
1
  ---
2
- license: apache-2.0
3
- language:
4
- - en
5
- - zh
6
  base_model:
7
  - alibaba-pai/Wan2.1-Fun-1.3B-Control
8
  - alibaba-pai/Wan2.1-Fun-14B-Control
9
- pipeline_tag: text-to-video
 
 
 
 
10
  ---
11
 
12
  # <center>💡Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models </center>
13
 
14
  <div style="display: flex; justify-content: center; gap: 5px;">
15
  <a href="https://lumen-relight.github.io"><img src="https://img.shields.io/badge/Project-Lumen-blue" alt="Project"></a>
16
- <a href="https://arxiv.org/abs/2508.12945"><img src="https://img.shields.io/badge/arXiv-Paper-red" alt="arXiv"></a>
17
  <a href="https://github.com/Kunbyte-AI/Lumen"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a>
18
  <a href="https://huggingface.co/Kunbyte/Lumen"><img src="https://img.shields.io/badge/🤗%20HF-Model-yellow" alt="HuggingFace"></a>
19
  <a href="https://huggingface.co/spaces/Kunbyte/Lumen"><img src="https://img.shields.io/badge/🤗%20HF-Space-yellow" alt="HuggingFace"></a>
20
  </div>
21
 
22
- 💡**Lumen** is a video relighting model that can relight the foreground and replace the background of a video base on the input text.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
23
 
24
- This repository contains the weights of **Lumen**. For more instructions about how to use our model, please refer to our [Github repository](https://github.com/Kunbyte-AI/Lumen).
 
 
1
  ---
 
 
 
 
2
  base_model:
3
  - alibaba-pai/Wan2.1-Fun-1.3B-Control
4
  - alibaba-pai/Wan2.1-Fun-14B-Control
5
+ language:
6
+ - en
7
+ - zh
8
+ license: apache-2.0
9
+ pipeline_tag: image-to-video
10
  ---
11
 
12
  # <center>💡Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models </center>
13
 
14
  <div style="display: flex; justify-content: center; gap: 5px;">
15
  <a href="https://lumen-relight.github.io"><img src="https://img.shields.io/badge/Project-Lumen-blue" alt="Project"></a>
16
+ <a href="https://huggingface.co/papers/2508.12945"><img src="https://img.shields.io/badge/arXiv-Paper-red" alt="arXiv"></a>
17
  <a href="https://github.com/Kunbyte-AI/Lumen"><img src="https://img.shields.io/badge/GitHub-Code-black" alt="GitHub"></a>
18
  <a href="https://huggingface.co/Kunbyte/Lumen"><img src="https://img.shields.io/badge/🤗%20HF-Model-yellow" alt="HuggingFace"></a>
19
  <a href="https://huggingface.co/spaces/Kunbyte/Lumen"><img src="https://img.shields.io/badge/🤗%20HF-Space-yellow" alt="HuggingFace"></a>
20
  </div>
21
 
22
+ 💡**Lumen** is an end-to-end video relighting framework developed on large-scale video generative models. It can relight the foreground and replace the background of a video based on flexible textual descriptions for instructing the control of lighting and background.
23
+
24
+ ## Introduction
25
+ Video relighting aims to replace the background in videos while correspondingly adjusting the lighting in the foreground with harmonious blending. Lumen preserves the original properties of the foreground (e.g., albedo) and propagates consistent relighting across temporal frames. It is trained on a large-scale dataset featuring a mixture of realistic and synthetic videos, utilizing a domain-aware adapter to decouple the learning of relighting and domain appearance distribution.
26
+
27
+ ### Authors
28
+ Jianshu Zeng, Yuxuan Liu, Yutong Feng, Chenxuan Miao, Zixiang Gao, Jiwang Qu, Jianzhang Zhang, Bin Wang, Kun Yuan.
29
+
30
+ ## 🚀 Quick Start
31
+
32
+ This repository contains the weights of **Lumen**. For more instructions about how to use the model, please refer to the [official GitHub repository](https://github.com/Kunbyte-AI/Lumen).
33
+
34
+ ### Environment Setup
35
+ ```bash
36
+ conda create -n lumen python=3.10 -y
37
+ conda activate lumen
38
+ pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu124
39
+ pip install -r requirements.txt
40
+ ```
41
+
42
+ ### Inference
43
+ After downloading the model weights and necessary base models (Wan2.1-Fun), you can run inference or the Gradio app:
44
+ ```bash
45
+ # Run text-to-video inference
46
+ python infer_t2v.py
47
+
48
+ # Launch the Gradio demo
49
+ python app_lumen.py
50
+ ```
51
+
52
+ ## 📋 Citation
53
+ If you find our work helpful, please consider citing:
54
+ ```bibtex
55
+ @article{zeng2025lumen,
56
+ title={Lumen: Consistent Video Relighting and Harmonious Background Replacement with Video Generative Models},
57
+ author={Zeng, Jianshu and Liu, Yuxuan and Feng, Yutong and Miao, Chenxuan and Gao, Zixiang and Qu, Jiwang and Zhang, Jianzhang and Wang, Bin and Yuan, Kun},
58
+ journal={arXiv preprint arXiv:2508.12945},
59
+ year={2025},
60
+ url={https://arxiv.org/abs/2508.12945},
61
+ }
62
+ ```
63
 
64
+ ## Acknowledgements
65
+ We would like to thank the contributors to [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), [VideoX-Fun](https://github.com/aigc-apps/VideoX-Fun), and the [Wan2.1](https://github.com/Wan-Video/Wan2.1) team for their open research and exploration.