LumaFlux v2 adapters: main (mixed corpus) and HDRTV1K in-domain

README.md

@@ -0,0 +1,143 @@
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
+base_model: black-forest-labs/FLUX.1-dev
+tags:
+  - inverse-tone-mapping
+  - hdr
+  - sdr-to-hdr
+  - diffusion
+  - flux
+  - video
+  - image-to-image
+pipeline_tag: image-to-image
+library_name: safetensors
+---
+
+# LumaFlux: Lifting 8-Bit Worlds to HDR Reality with Physically-Guided Diffusion Transformers
+
+[Paper (arXiv:2604.02787)](https://arxiv.org/abs/2604.02787)  · 
+[Code (GitHub)](https://github.com/shreshthsaini/LumaFlux)  · 
+[Project page](https://shreshthsaini.github.io/LumaFlux/)
+
+Shreshth Saini, Hakan Gedik, Neil Birkbeck, Yilin Wang, Balu Adsumilli, Alan C. Bovik
+— The University of Texas at Austin and Google, Inc.
+
+LumaFlux converts 8-bit SDR (BT.709) images and video into 10-bit HDR
+(PQ, BT.2020) by adapting a **frozen** FLUX.1-dev MM-DiT. This repository holds
+the trained **adapter weights only**: 71,315,893 parameters, 0.57% of the
+backbone. The frozen FLUX.1-dev backbone and SigLIP encoder are downloaded
+separately at load time.
+
+Conversion is prompt-free. Inference integrates a rectified-flow bridge from the
+SDR latent in 8 steps, and video is handled by the same image model with no
+temporal layers, optical flow, or video fine-tuning.
+
+## Checkpoints
+
+| File | Training corpus | Steps | Intended use |
+|---|---|---|---|
+| `lumaflux-main.safetensors` | Mixed UGC + PGC (314,396 pairs from 2,092 videos) | 100,000 | **Default.** General SDR-to-HDR conversion, video, and the Luma-Eval benchmark. |
+| `lumaflux-hdrtv1k.safetensors` | HDRTV1K training split | 50,000 | In-domain HDRTV1K comparisons and compression-robustness studies. |
+
+Both files carry the identical architecture and are interchangeable at load
+time; they differ only in training data and schedule.
+
+## Results
+
+**Luma-Eval** (our common run-by-us protocol; 100 held-out frames, synthetic
+tone-mapping and codec degradations). `lumaflux-main`:
+
+| Method | PU21-PSNR ↑ | PU21-SSIM ↑ | ΔE_ITP ↓ | HDR-VDP-3 ↑ | FR-HIDRO ↓ |
+|---|---:|---:|---:|---:|---:|
+| BT.2446c inverse | 23.74 | 0.8291 | 61.60 | 5.611 | 0.753 |
+| HDRTVNet++ | 23.03 | 0.8192 | 66.67 | 5.417 | 0.703 |
+| FMNet | 22.86 | 0.8176 | 67.92 | 5.412 | 0.724 |
+| **LumaFlux** | **24.23** | **0.8294** | **56.54** | **5.812** | **0.631** |
+
+**HDRTV1K** (117 published test pairs, evaluated under the authors' protocol):
+
+| Model | PSNR ↑ | SSIM ↑ | SR-SIM ↑ | ΔE_ITP ↓ | HDR-VDP-3 ↑ |
+|---|---:|---:|---:|---:|---:|
+| `lumaflux-hdrtv1k` (in-domain) | 33.34 | 0.9427 | 0.9941 | 14.69 | 7.962 |
+| `lumaflux-main` (zero-shot) | 27.52 | 0.9307 | 0.9837 | 29.86 | 7.618 |
+
+The zero-shot row is reported for transparency: the main model never sees the
+HDRTV1K training split, so it is not comparable to methods trained on it.
+
+**Video.** Sharing one bridge-noise realization across frames and smoothing the
+applied tone-curve parameters with an EMA reduces signed excess PU21 flicker by
+**53.3%**, with pooled temporal energy within 3.3% of the reference, without any
+temporal training.
+
+**Cost.** 8 solver steps, 5.28 s per 1080p frame and 27.1 GB peak memory on a
+single GH200. Forty steps cost 4.7x more without improving quality.
+
+## Usage
+
+```bash
+pip install "lumaflux @ git+https://github.com/shreshthsaini/LumaFlux.git"
+huggingface-cli download shreshthsaini/LumaFlux lumaflux-main.safetensors --local-dir weights
+```
+
+```bash
+# Image or video in, 10-bit PQ/BT.2020 HDR out
+python -m lumaflux.inference.cli \
+  --config configs/model/flux_dev.yaml \
+  --adapters weights/lumaflux-main.safetensors \
+  --input input_sdr.mp4 --output output_hdr.mp4 --steps 8
+```
+
+Loading the adapters also pulls `black-forest-labs/FLUX.1-dev` (gated; accept
+its license first) and `google/siglip-so400m-patch14-384`. Full instructions,
+data preparation, training, and the evaluation suite are in the
+[GitHub repository](https://github.com/shreshthsaini/LumaFlux).
+
+## Training
+
+Both checkpoints were trained with a frozen backbone: only PGA (gated low-rank
+attention residuals), PCM (SigLIP FiLM conditioning), the HDR Residual Coupler,
+the timestep-layer conditioner, and the monotone RQS tone head receive
+gradients. Optimization used AdamW at learning rate 2e-4, global batch 16, 512px
+crops, bf16, and a rectified-flow velocity objective with decoded-reconstruction
+and spline-smoothness terms. The main model trained for about 36 hours on four
+GH200 GPUs (~144 GPU-hours).
+
+Training data combines CHUG user-generated HDR, LIVE-TMHDR (including
+expert-graded SDR), HDRTV1K, and the Netflix Open Content *Sol Levante* HDR10
+master. SDR inputs are either genuine expert grades or generated with a frozen
+degradation chain of eight tone-mapping operators, BT.2020-to-BT.709 gamut
+conversion, and three CRF settings.
+
+## License and intended use
+
+These adapters only function on top of FLUX.1-dev and are therefore released
+under the **FLUX.1-dev Non-Commercial License**; research use only. The
+accompanying source code is Apache-2.0. Users must separately accept the
+FLUX.1-dev license.
+
+Intended for research on inverse tone mapping, archival SDR remastering, and
+HDR video pipelines. Not validated for medical, forensic, or safety-critical
+imaging.
+
+## Limitations
+
+- The frozen VAE was not trained as an HDR codec, so the tone head calibrates
+  its output but cannot recover detail absent from heavily clipped SDR.
+- Temporal stabilization is inference-time only and does not model long-range
+  motion explicitly.
+- On the selected qualitative scenes the outputs measure slightly more saturated
+  than their references (mean ITP chroma 60.65 versus 49.80).
+- Correct display metadata and EOTF settings are required at playback.
+
+## Citation
+
+```bibtex
+@article{saini2026lumaflux,
+  title   = {LumaFlux: Lifting 8-Bit Worlds to HDR Reality with Physically-Guided Diffusion Transformers},
+  author  = {Saini, Shreshth and Gedik, Hakan and Birkbeck, Neil and Wang, Yilin and Adsumilli, Balu and Bovik, Alan C.},
+  journal = {arXiv preprint arXiv:2604.02787},
+  year    = {2026}
+}
+```

configs/hdrtv1k.yaml

@@ -0,0 +1,39 @@
+model:
+  backbone: black-forest-labs/FLUX.1-dev
+  siglip: google/siglip-so400m-patch14-384
+  dtype: bf16
+  gradient_checkpointing: false
+  rank: 8
+  bottleneck: 64
+  num_knots: 8
+  phys_channels: 32
+  stats_dim: 16
+  num_bands: 8
+  num_null_tokens: 8
+  modulation_hidden: 128
+  guidance_scale: 1.0
+  peak_nits: 1000.0
+train:
+  max_steps: 50000
+  batch_size: 2
+  grad_accum: 2
+  lr: 0.0002
+  betas:
+  - 0.9
+  - 0.999
+  weight_decay: 0.01
+  warmup_steps: 5000
+  crop_size: 512
+  bridge_noise: 0.05
+  recon_every: 4
+  w_velocity: 1.0
+  lambda1: 1.0
+  lambda2: 0.5
+  lambda3: 0.1
+  max_grad_norm: 1.0
+  mixed_precision: bf16
+  num_workers: 4
+  log_every: 50
+  ckpt_every: 5000
+  trackio_project: lumaflux
+  seed: 0

configs/main.yaml

@@ -0,0 +1,46 @@
+# Full v2 training run. The 4-node launcher gives an effective global batch
+# of 4 nodes x 2 samples x 2 accumulation steps = 16.
+model:
+  backbone: black-forest-labs/FLUX.1-dev
+  siglip: google/siglip-so400m-patch14-384
+  dtype: bf16
+  gradient_checkpointing: false
+  rank: 8
+  bottleneck: 64
+  num_knots: 8
+  phys_channels: 32
+  stats_dim: 16
+  num_bands: 8
+  num_null_tokens: 8
+  modulation_hidden: 128
+  guidance_scale: 1.0
+  peak_nits: 1000.0
+
+data:
+  manifest: data/curated/manifest.jsonl  # path produced by scripts/prepare_data.py
+
+train:
+  max_steps: 100000
+  batch_size: 2
+  grad_accum: 2
+  lr: 2.0e-4
+  betas: [0.9, 0.999]
+  weight_decay: 0.01
+  warmup_steps: 5000
+  scheduler: cosine
+  crop_size: 512
+  bridge_noise: 0.05
+  recon_every: 4
+  w_velocity: 1.0
+  lambda1: 1.0
+  lambda2: 0.5
+  lambda3: 0.1
+  max_grad_norm: 1.0
+  mixed_precision: bf16
+  num_workers: 8
+  log_every: 50
+  ckpt_every: 5000
+  ckpt_keep: 3
+  trackio_project: lumaflux
+  run_name: lumaflux-main-v2
+  seed: 0

lumaflux-hdrtv1k.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:758160a9dec71027b3a3f2f33111d70ddc62d72b1c91b57a3fbc04fa57f89b0a
+size 285366796

lumaflux-main.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c22693b09d769526fecdd7dd6d0b0bcaae86dbf7d3d8c79d7ae66b7f722282e1
+size 285366796