Add paper link, project page, and GitHub repository

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +27 -12
README.md CHANGED
@@ -1,21 +1,23 @@
1
  ---
2
  license: apache-2.0
3
- tags:
4
- - video
5
- - relighting
6
- - inverse-rendering
7
- - diffusion
8
- - cosmos
9
  pipeline_tag: image-to-image
 
 
 
 
 
 
10
  ---
11
 
12
  # World Inverse Renderer
13
 
14
- Video inverse rendering model based on NVIDIA Cosmos 7B video diffusion transformer, fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file).
 
 
15
 
16
  ## Model Description
17
 
18
- This model performs **inverse rendering** on images and videos: given an input RGB frame, it estimates physically-based G-buffer maps:
19
 
20
  - **Basecolor** (albedo)
21
  - **Normal** (surface normals)
@@ -23,16 +25,18 @@ This model performs **inverse rendering** on images and videos: given an input R
23
  - **Roughness**
24
  - **Metallic**
25
 
26
- These G-buffers can then be used with a forward renderer to relight the scene under arbitrary environment lighting (HDRI maps).
27
 
28
  ## Architecture
29
 
30
- - Based on NVIDIA Cosmos 7B video diffusion transformer
31
- - Fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file)
32
- - Supports both single-image and multi-frame video inverse rendering
33
 
34
  ## Usage
35
 
 
 
36
  ```bash
37
  # Inverse rendering on images
38
  CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
@@ -52,3 +56,14 @@ CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/infer
52
  - Python 3.10
53
  - NVIDIA GPU with >= 16GB VRAM (48GB+ recommended)
54
  - CUDA 12.0+
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
3
  pipeline_tag: image-to-image
4
+ tags:
5
+ - video
6
+ - relighting
7
+ - inverse-rendering
8
+ - diffusion
9
+ - cosmos
10
  ---
11
 
12
  # World Inverse Renderer
13
 
14
+ [**Project Page**](https://alaya-studio.github.io/renderer) | [**Paper**](https://huggingface.co/papers/2604.02329) | [**GitHub**](https://github.com/ShandaAI/AlayaRenderer)
15
+
16
+ This repository contains the **World Inverse Renderer** model presented in the paper [Generative World Renderer](https://huggingface.co/papers/2604.02329).
17
 
18
  ## Model Description
19
 
20
+ This model performs **inverse rendering** on images and videos. Based on the NVIDIA Cosmos 7B video diffusion transformer, it has been fine-tuned on a large-scale dataset of 4M frames extracted from AAA games. Given an input RGB frame or video, it estimates physically-based G-buffer maps:
21
 
22
  - **Basecolor** (albedo)
23
  - **Normal** (surface normals)
 
25
  - **Roughness**
26
  - **Metallic**
27
 
28
+ These G-buffers can subsequently be used with a forward renderer to relight or edit the scene under arbitrary lighting conditions.
29
 
30
  ## Architecture
31
 
32
+ - Based on NVIDIA Cosmos 7B video diffusion transformer.
33
+ - Fine-tuned on the AlayaRenderer dataset (4M+ frames from AAA games like Cyberpunk 2077 and Black Myth: Wukong).
34
+ - Supports both single-image and multi-frame video inverse rendering.
35
 
36
  ## Usage
37
 
38
+ For environment setup and full inference code, please refer to the [official GitHub repository](https://github.com/ShandaAI/AlayaRenderer).
39
+
40
  ```bash
41
  # Inverse rendering on images
42
  CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
 
56
  - Python 3.10
57
  - NVIDIA GPU with >= 16GB VRAM (48GB+ recommended)
58
  - CUDA 12.0+
59
+
60
+ ## Citation
61
+
62
+ ```bibtex
63
+ @article{huang2026generativeworldrenderer,
64
+ title={Generative World Renderer},
65
+ author={Zheng-Hui Huang and Zhixiang Wang and Jiaming Tan and Ruihan Yu and Yidan Zhang and Bo Zheng and Yu-Lun Liu and Yung-Yu Chuang and Kaipeng Zhang},
66
+ journal={arXiv preprint arXiv:2604.02329},
67
+ year={2026}
68
+ }
69
+ ```