File size: 1,816 Bytes
ff3bcc2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 | ---
license: apache-2.0
tags:
- video
- relighting
- inverse-rendering
- diffusion
- cosmos
pipeline_tag: image-to-image
---
# World Inverse Renderer
Video inverse rendering model based on NVIDIA Cosmos 7B video diffusion transformer, fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file).
## Model Description
This model performs **inverse rendering** on images and videos: given an input RGB frame, it estimates physically-based G-buffer maps:
- **Basecolor** (albedo)
- **Normal** (surface normals)
- **Depth**
- **Roughness**
- **Metallic**
These G-buffers can then be used with a forward renderer to relight the scene under arbitrary environment lighting (HDRI maps).
## Architecture
- Based on NVIDIA Cosmos 7B video diffusion transformer
- Fine-tuned on [custom dataset](https://github.com/ShandaAI/AlayaRenderer?tab=readme-ov-file)
- Supports both single-image and multi-frame video inverse rendering
## Usage
```bash
# Inverse rendering on images
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
--checkpoint_dir checkpoints --diffusion_transformer_dir Diffusion_Renderer_Inverse_Cosmos_7B \
--dataset_path=your_input_images/ --num_video_frames 1 --group_mode webdataset \
--video_save_folder=output/ --save_video=False
# Inverse rendering on video frames
CUDA_HOME=$CONDA_PREFIX PYTHONPATH=$(pwd) python cosmos_predict1/diffusion/inference/inference_inverse_renderer.py \
--checkpoint_dir checkpoints --diffusion_transformer_dir Diffusion_Renderer_Inverse_Cosmos_7B \
--dataset_path=your_video_frames/ --num_video_frames 57 \
--video_save_folder=output/
```
## Requirements
- Python 3.10
- NVIDIA GPU with >= 16GB VRAM (48GB+ recommended)
- CUDA 12.0+
|