Sync model card with GitHub README

README.md

@@ -22,11 +22,11 @@ tags:
 
 [![Project Page](https://img.shields.io/badge/Project-Page-yellow?logo=googlechrome&logoColor=yellow)](https://eyeline-labs.github.io/DiffHDR/)
 [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b?logo=arxiv&logoColor=red)](https://arxiv.org/abs/2604.06161)
-[![Code](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/Eyeline-Labs/DiffHDR)
+[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DiffHDR-blue)](https://huggingface.co/ZhengmingYu/DiffHDR)
 [![Demo Video](https://img.shields.io/badge/Demo-Video-red?logo=youtube&logoColor=white)](https://youtu.be/kq8qZfwBRs0)
 
 [Zhengming Yu](https://yzmblog.github.io/)<sup>1,2</sup>, [Li Ma](https://limacv.github.io/homepage/)<sup>2</sup>, [Mingming He](https://mingminghe.com/)<sup>2</sup>, [Leo Isikdogan](https://isikdogan.com/)<sup>3</sup>, [Yuancheng Xu](https://yuancheng-xu.github.io/)<sup>2,3</sup>, [Dmitriy Smirnov](https://dsmirnov.com/)<sup>3</sup>, [Pablo Salamanca](https://pablosalaman.ca/)<sup>2,3</sup>, [Dao Mi](#)<sup>3</sup>, [Pablo Delgado](#)<sup>3</sup>, [Ning Yu](https://ningyu1991.github.io/)<sup>2,3</sup>, [Julien Philip](https://julienphilip.com/)<sup>2</sup>, [Xin Li](https://people.tamu.edu/~xinli/)<sup>1</sup>, [Wenping Wang](https://engineering.tamu.edu/cse/profiles/Wang-Wenping.html)<sup>1</sup>, [Paul Debevec](https://www.debevec.org/)<sup>2,3</sup><br/>
-<sup>1</sup>Texas A&amp;M University, <sup>2</sup>Eyeline Labs, <sup>3</sup>Netflix
+<sup>1</sup>Texas A&amp;M University, <sup>2</sup>Eyeline Labs, <sup>3</sup>Netflix<br/>
 
 <p align="center">
   <img src="./assets/teaser.png" alt="DiffHDR teaser figure">
@@ -34,64 +34,84 @@ tags:
 
 ## 📝 Abstract
 
-> Most digital videos are stored in 8-bit low dynamic range (LDR) formats, where much of the original high dynamic range (HDR) scene radiance is lost due to saturation and quantization. This loss of highlight and shadow detail precludes mapping accurate luminance to HDR displays and limits meaningful re-exposure in post-production workflows. Although techniques have been proposed to convert LDR images to HDR through dynamic range expansion, they struggle to restore realistic detail in the over- and underexposed regions. To address this, we present **DiffHDR**, a framework that formulates LDR-to-HDR conversion as a generative radiance inpainting task within the latent space of a video diffusion model. By operating in Log-Gamma color space, DiffHDR leverages spatio-temporal generative priors from a pretrained video diffusion model to synthesize plausible HDR radiance in over- and underexposed regions while recovering the continuous scene radiance of the quantized pixels. Our framework further enables controllable LDR-to-HDR video conversion guided by text prompts or reference images. To address the scarcity of paired HDR video data, we develop a pipeline that synthesizes high-quality HDR video training data from static HDRI maps. Extensive experiments demonstrate that DiffHDR significantly outperforms state-of-the-art approaches in radiance fidelity and temporal stability, producing realistic HDR videos with considerable latitude for re-exposure.
+> Most digital videos are stored in 8-bit low dynamic range (LDR) formats, where much of the original high dynamic range (HDR) scene radiance is lost due to saturation and quantization. This loss of highlight and shadow detail precludes mapping accurate luminance to HDR displays and limits meaningful re-exposure in post-production workflows. Although techniques have been proposed to convert LDR images to HDR through dynamic range expansion, they struggle to restore realistic detail in over- and underexposed regions. To address this, we present **DiffHDR**, a framework that formulates LDR-to-HDR conversion as a generative radiance inpainting task in the latent space of a video diffusion model. By operating in Log-Gamma color space, DiffHDR leverages spatio-temporal generative priors from a pretrained video diffusion model to synthesize plausible HDR radiance in over- and underexposed regions while recovering the continuous scene radiance. Our framework further enables controllable LDR-to-HDR video conversion guided by text prompts or reference images. To address the scarcity of paired HDR video data, we develop a pipeline that synthesizes high-quality HDR video training data from static HDRI maps. Extensive experiments demonstrate that DiffHDR significantly outperforms state-of-the-art approaches in radiance fidelity and temporal stability, producing realistic HDR videos with considerable latitude for re-exposure.
 
-## 🔬 Model overview
-
-DiffHDR uses a proposed **Log-Gamma Color Mapping** to compress linear HDR
-radiance into a bounded representation aligned with natural LDR statistics.
-This allows the pretrained LDR video VAE to encode and decode HDR content
-without VAE finetuning or shifting its learned latent space. A LoRA-adapted
-[Wan2.1-VACE-14B](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B) backbone then
-performs generative radiance inpainting conditioned on the input LDR content
-and exposure masks, with optional text or reference-image control. The decoded
-result is inverse Log-Gamma mapped to linear HDR radiance.
-
-## 📦 Model files
-
-This repository contains LoRA adapters, not a standalone model. Download the
-base model separately before running inference.
-
-| File | Intended inference entry points |
-| --- | --- |
-| `DiffHDR.safetensors` | `infer_video.py`, `infer_image.py`, and `infer_long_video.py` |
-| `DiffHDR_Pano.safetensors` | `infer_hdri.py` for 360-degree HDR panoramas |
 
 ## 🛠️ Setup
 
-Clone and install the inference code:
-
 ```bash
-git clone https://github.com/Eyeline-Labs/DiffHDR.git
-cd DiffHDR
-
 conda create -n diffhdr python=3.10 -y
 conda activate diffhdr
 
+# Install PyTorch (CUDA 11.8)
 pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 \
     --index-url https://download.pytorch.org/whl/cu118
+
+# Install DiffHDR
+cd /path/to/DiffHDR_Code
 pip install -e .
 pip install -r requirements.txt
 ```
 
-Download the base model and the DiffHDR adapters:
+### Base model
+
+Download Wan2.1-VACE-14B (~75 GB). This single repo contains everything needed --
+the 7 DiT shards, the T5 text encoder, the VAE, and the umt5-xxl tokenizer:
 
 ```bash
-hf download Wan-AI/Wan2.1-VACE-14B \
-    --local-dir models/Wan-AI/Wan2.1-VACE-14B
+hf download Wan-AI/Wan2.1-VACE-14B --local-dir models/Wan-AI/Wan2.1-VACE-14B
+```
 
+Expected layout:
+
+```
+models/Wan-AI/Wan2.1-VACE-14B/
+├── diffusion_pytorch_model-0000{1..7}-of-00007.safetensors
+├── models_t5_umt5-xxl-enc-bf16.pth
+├── Wan2.1_VAE.pth
+└── google/umt5-xxl/
+```
+
+Set the `MODEL_BASE` environment variable to point the scripts at a different
+root instead of `models/`.
+
+### LoRA checkpoints
+
+```bash
+mkdir -p models
 hf download ZhengmingYu/DiffHDR --local-dir models
 ```
 
-The Wan2.1-VACE-14B download is approximately 75 GB and is not included in this
-repository. Set `MODEL_BASE` if you store the base model somewhere other than
-`models/`.
+This fetches both LoRA weights (58 MB each) into `models/`:
+
+| File | Use with |
+|------|----------|
+| `DiffHDR.safetensors` | `infer_video.py`, `infer_image.py`, `infer_long_video.py` |
+| `DiffHDR_Pano.safetensors` | `infer_hdri.py` (360 panoramas) |
+
+### Optional: Flash Attention
+
+Not required -- all inference paths fall back to PyTorch SDPA. Install it only if
+you want the speedup, and note that it compiles CUDA kernels from source (needs
+`nvcc`, takes tens of minutes):
+
+```bash
+pip install psutil ninja packaging wheel   # flash_attn's setup.py needs these
+pip install flash_attn --no-build-isolation
+```
+
 
 ## 🎥 Inference
 
-Minimal video example:
+Our paper results were produced with the default `--num_inference_steps 50`. In
+practice we found that 10 steps gives comparable quality on many cases, so the
+video demo commands below pass `--num_inference_steps 10` to keep them fast.
+Drop that flag to reproduce the paper setting.
+
+### Video (MP4 or image folder)
 
 ```bash
+# From MP4 file
 python infer_video.py \
     --lora_path models/DiffHDR.safetensors \
     --input_path demo/room_window.mp4 \
@@ -101,17 +121,51 @@ python infer_video.py \
     --add_mask --use_under_exposure_mask --crop_and_resize --srgb_to_lg
 ```
 
-Additional entry points:
+#### Text-conditioned inference
+
+Provide a descriptive prompt to guide HDR reconstruction:
+
+```bash
+python infer_video.py \
+    --lora_path models/DiffHDR.safetensors \
+    --input_path demo/wooden_house \
+    --output_dir results/text_cond \
+    --num_inference_steps 10 \
+    --prompt "over-exposed: A bright ocean landscape visible through the skylight window, with a wide blue sea stretching to the horizon and soft clouds in the sky. Sunlight shines through the window and softly illuminates the wooden attic interior while keeping the indoor scene unchanged." \
+    --seed 33 \
+    --add_mask --crop_and_resize --srgb_to_lg
+```
+
+#### Image-conditioned inference
+
+Provide a reference image to guide the style and tone of the HDR output:
+
+```bash
+python infer_video.py \
+    --lora_path models/DiffHDR.safetensors \
+    --input_path demo/wooden_house \
+    --output_dir results/image_cond \
+    --reference_image_path demo/ref_gemini_city.jpg \
+    --prompt "" \
+    --num_inference_steps 10 \
+    --add_mask --crop_and_resize --srgb_to_lg
+```
+
+### Single Image
 
 ```bash
-# Single image
 python infer_image.py \
     --lora_path models/DiffHDR.safetensors \
     --input_path demo/sample_image.png \
     --output_dir results/image_output \
     --prompt ""
+```
+
+### Long Video (sliding window)
 
-# Long video using overlapping temporal windows
+For videos with more than 33 frames:
+
+```bash
 python infer_long_video.py \
     --lora_path models/DiffHDR.safetensors \
     --input_path demo/long_video_frames \
@@ -120,66 +174,71 @@ python infer_long_video.py \
     --window_size 33 --window_stride 16 \
     --use_prev_window_reference \
     --add_mask --crop_and_resize --srgb_to_lg
+```
+
+**How it works:**
+- Processes the video in overlapping windows of `window_size` frames
+- Stride of `window_stride` frames between windows (overlap = window_size - window_stride)
+- Linear temporal blending in overlap regions for smooth transitions
+- `--use_prev_window_reference`: passes a reference frame from the previous window for temporal consistency
+
+### HDRI Panorama
 
-# 360-degree LDR panorama to HDR panorama
+For single LDR panorama images (We extend this work to [HDRI](https://eyeline-labs.github.io/HDRI/)):
+
+```bash
 python infer_hdri.py \
     --lora_path models/DiffHDR_Pano.safetensors \
     --input_path demo/sample_pano.png \
     --output_dir results/hdri_output
 ```
 
-The inference scripts write linear HDR OpenEXR output. Video inference writes
-one EXR file per frame; panorama inference writes `predicted.exr`. See the
-[code repository README](https://github.com/Eyeline-Labs/DiffHDR#inference) for
-text conditioning, reference-image conditioning, arguments, and additional
-examples.
-
-## 🎯 Intended use
-
-DiffHDR is intended for research and creative LDR-to-HDR reconstruction,
-including:
-
-- reconstructing HDR video or still images from LDR input;
-- recovering plausible highlight and shadow content for display and
-  post-production workflows;
-- text- or reference-image-guided HDR reconstruction; and
-- reconstructing HDR environment panoramas.
-
-## ⚠️ Limitations and responsible use
-
-- Detail in clipped or quantized regions is generated by the model. It is a
-  plausible reconstruction and is not guaranteed to reproduce the original
-  scene radiance.
-- Outputs may contain hallucinated detail, temporal inconsistency, color
-  shifts, or exposure artifacts, especially on inputs outside the training
-  distribution.
-- Results can vary with prompts, reference images, random seeds, and inference
-  settings.
-- The model inherits limitations and potential biases from its base model and
-  training data.
-- Do not use generated output as a calibrated radiometric measurement, as
-  forensic evidence, or in safety-critical decisions.
-- Users are responsible for ensuring that their input media and intended use
-  comply with applicable rights, licenses, and laws.
-
-## 🏋️ Training and evaluation
-
-DiffHDR is trained as a LoRA adapter on top of Wan2.1-VACE-14B while keeping
-the video VAE frozen. Its synthetic training set is rendered from approximately
-800 high-resolution HDRIs and contains about 5,400 temporally consistent
-81-frame HDR sequences. Exposure shifts, temporally correlated camera noise,
-clipping, and 8-bit quantization are simulated to produce paired LDR inputs.
-Method details, experimental settings, and comparisons are reported in the
-[paper](https://arxiv.org/abs/2604.06161).
+This uses overexposure mask detection (luma + channel clipping) and outputs a single HDR EXR panorama at 1024x2048 by default.
 
-## 📄 License
+## 📊 Eval
+Evaluate generated HDR EXR frames using `eval/cal_sample.py`:
 
-The DiffHDR LoRA adapter weights in this repository are released under the
-[Apache License 2.0](LICENSE).
+```bash
+# NR metrics only (MUSIQ, CLIPIQA, PU21-PIQE):
+python eval/cal_sample.py \
+    --gen_dir results/video_mp4 \
+    --out_csv results/video_mp4_eval.csv
+
+# With ground truth (adds FovVideoVDP):
+python eval/cal_sample.py \
+    --gen_dir results/video_mp4 \
+    --gt_dir /path/to/gt_exr_frames \
+    --out_csv results/video_mp4_eval.csv
+
+# With DOVER video quality metric:
+python eval/cal_sample.py \
+    --gen_dir results/video_mp4 \
+    --out_csv results/video_mp4_eval.csv \
+    --dover_repo /path/to/DOVER
+```
+
+| Metric | Type | Description |
+|--------|------|-------------|
+| MUSIQ | NR | No-reference image quality (tonemapped) |
+| CLIPIQA | NR | CLIP-based image quality (tonemapped) |
+| PU21-PIQE | NR | Perceptual quality on PU21-encoded HDR luminance |
+| FovVideoVDP | FR | Full-reference HDR visual difference (JOD) |
+| DOVER | NR | No-reference video quality (tonemapped MP4) |
+| FID | FR | Distribution distance on tonemapped patches |
+
+For HDR-VDP-3, we follow LEDiff to use the Matlab scripts, please refer the `run_hdrvdp3_dir.m` for the configuration details.
+
+
+## 🏋️ Training
+
+```bash
+# Launch LoRA training
+bash scripts/train.sh
+```
+
+The training script uses HuggingFace Accelerate for distributed training.
 
-The Wan2.1-VACE-14B base weights are distributed separately and remain subject
-to their own license and model-card guidance. This license does not grant rights
-to third-party code, datasets, or user-supplied input content.
+**Training data format:** EXR frames organized by the metadata CSV, with sRGB LDR and linear HDR pairs.
 
 ## 📚 Citation
 
@@ -192,9 +251,25 @@ to third-party code, datasets, or user-supplied input content.
 }
 ```
 
+
 ## 🙏 Acknowledgements
+Our work is built upon many awesome prior works:
+
+- **[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio)** --
+  the `diffsynth/` package in this repository is a reduced, modified fork of it.
+- **[Wan2.1-VACE-14B](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B)** --
+  the base video diffusion model that our LoRA is trained on top of.
+  
+We thank these authors for their great works and open-source contribution.
+
+
+
+
+## 📄 License
+
+This project is released under the licence in [LICENSE](LICENSE).
 
-DiffHDR builds on
-[Wan2.1-VACE-14B](https://huggingface.co/Wan-AI/Wan2.1-VACE-14B) and
-[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio). Please also
-credit and follow the license terms of these upstream projects.
+It bundles third-party code: `diffsynth/` is derived from
+[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio), licensed under
+Apache-2.0. Files in that directory have been modified from the originals. The
+upstream copyright and licence terms continue to apply to them.