nviolante/dot3d

Lighting-Consistent Object Transfer Across Radiance Fields

Paper: PDF | Project Page: dot3d | HAL: hal-05657202

Authors: Nicolas Violante¹ · George Kopanas² · Linus Franke¹ · Julien Philip³ · George Drettakis¹

¹Inria, Université Côte d'Azur · ²Google DeepMind · ³Eyeline Labs

---

Model Description

DOT3D is a diffusion-based image harmonization model that corrects lighting inconsistencies when compositing objects from one scene into another. It is the 2D harmonization backbone of a complete 3D pipeline for transferring objects between 3D Gaussian Splatting (3DGS) captures.

When an object is extracted from a source scene and naively pasted into a target scene, the result is unrealistic due to mismatched lighting. DOT3D harmonizes each rendered view of the composite to match the target scene's lighting, and these views are then consolidated via 3DGS post-optimization.

The model is built on FLUX.1-schnell and fine-tuned on a heterogeneous dataset combining synthetic, generated, and real image pairs (inconsistent composite input → consistent output).

Two variants are available:

Variant	Inputs	Checkpoint
Image-Mask	Composite image + binary mask of inserted object	image-mask/
Image-Background	Composite image + original background	image-background/

---

Usage

Installation

git clone https://github.com/graphdeco-inria/dot3d
cd dot3d
conda create -n dot3d python=3.10 -y
conda activate dot3d
export CUDA_VERSION=cu121  # adjust to your system
bash install.sh

Image-Mask variant

from PIL import Image
from huggingface_hub import snapshot_download
from wrappers import DOT3DHarmonizationWrapper

snapshot_download(repo_id="nviolante/dot3d", allow_patterns="image-mask/*", local_dir="checkpoints")

wrapper = DOT3DHarmonizationWrapper("checkpoints/image-mask")

image = Image.open("composite.png").convert("RGB")
mask  = Image.open("mask.png").convert("RGB")

result = wrapper.predict_image(image, mask, num_inference_steps=4)
result["prediction"].save("harmonized.png")

Image-Background variant

snapshot_download(repo_id="nviolante/dot3d", allow_patterns="image-background/*", local_dir="checkpoints")

wrapper = DOT3DHarmonizationWrapper("checkpoints/image-background")

image      = Image.open("composite.png").convert("RGB")
background = Image.open("background.png").convert("RGB")

result = wrapper.predict_image(image, background, num_inference_steps=4)
result["prediction"].save("harmonized.png")

---

Training Data

The model was trained on a heterogeneous mixture of image pairs (inconsistent composite, consistent ground truth):

• Blender — synthetic renders with controlled relighting
• FLUX-generated — synthetically composited pairs produced with a generative model
• ORIDA — real image pairs from ORIDA with relighting supervision

Full dataset: nviolante/dot3d

---

Evaluation

Evaluated on PSNR, SSIM, LPIPS, FID, and KID on the test splits of all three datasets. Pre-computed 3D results are available at results_3d.

---

Hardware Requirements

Task	Hardware
Training	4× H100 (96 GB VRAM)
3D post-optimization	1× H100 (96 GB VRAM)
Inference (harmonization only)	Consumer GPU

---

BibTeX

@article{violante2026dot3d,
  author  = {Violante, Nicolás and Kopanas, George and Franke, Linus and Philip, Julien and Drettakis, George},
  title   = {Lighting-Consistent Object Transfer Across Radiance Fields},
  journal = {Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering)},
  year    = {2026},
  volume  = 45,
  number  = 4
}

---

Acknowledgments

Funded by the European Union ERC Advanced Grants NERPHYS (101141721) and EXPLORER (101097259). Experiments used the Grid'5000 testbed. The authors thank Adobe and NVIDIA for software and hardware donations.