AlbeRota commited on
Commit
a80c540
·
verified ·
1 Parent(s): 7933d2d

Upload weights, notebooks, sample images

Browse files
Files changed (1) hide show
  1. README.md +14 -15
README.md CHANGED
@@ -30,15 +30,15 @@ UnReflectAnything works on both natural indoor and surgical/endoscopic domain da
30
 
31
 
32
 
33
- * **<font color="#a001e0">Encoder</font> ($\mathit{\textcolor{a001e0}{E}}$ )**: Processes the input image $\mathbf{I}$ to extract a rich latent representation, $\mathbf{F}_\ell$. This is the off-the-shelf pretrained [DINOv3-large](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m)
34
 
35
- * **<font color="#0167ff">Reflection Predictor</font> ($\mathit{\textcolor{0167ff}{H}}$ )**: Predicts a soft highlight mask (**H**), identifying areas of specular highlights.
36
 
37
- * **Masking Operation</font> ($\mathit{P}$ )**: A binary mask **P** is derived from the prediction and applied to the feature map: $(1-\mathbf{P}) \odot \mathbf{F}_\ell$. This removes features contaminated by reflections, leaving "holes" in the data.
38
 
39
- * **<font color="#23ac2c">Token Inpainter</font> ($\mathit{\textcolor{23ac2c}{T}}$ )**: Acts as a neural in-painter. It processes the masked features and uses the surrounding clean context prior and a learned mask token to synthesize the missing information in embedding space, producing the completed feature map $\mathbf{F}_{\text{comp}}$.
40
 
41
- * **<font color="#ff7700">Decoder</font> ($\mathit{\textcolor{ff7700}{D}}$ )**: Project the completed features back into the pixel space to generate the final, reflection-free image $\mathbf{I}_{\text{diff}}$.
42
 
43
  ---
44
 
@@ -48,8 +48,8 @@ We train UnReflectAnything with **Synthetic Specular Supervision** by inferring
48
  ![SupervisionExamples](https://raw.githubusercontent.com/alberto-rota/UnReflectAnything/refs/heads/main/assets/highlights.png)
49
 
50
  We train the model in two stages
51
- 1. **DPT Decoder Pre-Training**: The **<font color="#ff7700">Decoder</font>** is first pre-trained in an autoencoder configuration ($\min_{\theta} \mathcal{L}(M_{\theta}(\mathbf{I}), \mathbf{I})$) to ensure it can reconstruct realistic RGB textures from the DINOV3 latent space.
52
- 2. **End-to-End Refinement**: The full pipeline is then trained to predict reflection masks from $\mathit{\textcolor{0167ff}{H}}$, and fill them using the **<font color="#38761D">Token Inpainter</font>**, ensuring the final output is both visually consistent and physically accurate. The decoder is also fine-tuned at this stage
53
 
54
 
55
 
@@ -103,14 +103,13 @@ Refer to the [Wiki](https://github.com/alberto-rota/UnReflectAnything/wiki) for
103
  If you use UnReflectAnything in your research or pipeline, please cite our paper:
104
 
105
  ```bibtex
106
- @misc{rota2025unreflectanythingrgbonlyhighlightremoval,
107
- title={UnReflectAnything: RGB-Only Highlight Removal by Rendering Synthetic Specular Supervision},
108
- author={Alberto Rota and Mert Kiray and Mert Asim Karaoglu and Patrick Ruhkamp and Elena De Momi and Nassir Navab and Benjamin Busam},
109
- year={2025},
110
- eprint={2512.09583},
111
- archivePrefix={arXiv},
112
- primaryClass={cs.CV},
113
- url={[https://arxiv.org/abs/2512.09583](https://arxiv.org/abs/2512.09583)},
114
  }
115
 
116
  ```
 
30
 
31
 
32
 
33
+ * **<font color="#a001e0">Encoder</font> (E)**: Processes the input image **I** to extract a rich latent representation, **F_l**. This is the off-the-shelf pretrained [DINOv3-large](https://huggingface.co/facebook/dinov3-vitl16-pretrain-lvd1689m)
34
 
35
+ * **<font color="#0167ff">Reflection Predictor</font> (H)**: Predicts a soft highlight mask (**H**), identifying areas of specular highlights.
36
 
37
+ * **Masking Operation (P)**: A binary mask **P** is derived from the prediction and applied to the feature map: `(1 - P) * F_l`. This removes features contaminated by reflections, leaving "holes" in the data.
38
 
39
+ * **<font color="#23ac2c">Token Inpainter</font> (T)**: Acts as a neural in-painter. It processes the masked features and uses the surrounding clean context prior and a learned mask token to synthesize the missing information in embedding space, producing the completed feature map **F_comp**.
40
 
41
+ * **<font color="#ff7700">Decoder</font> (D)**: Project the completed features back into the pixel space to generate the final, reflection-free image **I_diff**.
42
 
43
  ---
44
 
 
48
  ![SupervisionExamples](https://raw.githubusercontent.com/alberto-rota/UnReflectAnything/refs/heads/main/assets/highlights.png)
49
 
50
  We train the model in two stages
51
+ 1. **DPT Decoder Pre-Training**: The **<font color="#ff7700">Decoder</font>** is first pre-trained in an autoencoder configuration (minimize `L(M_θ(I), I)`) to ensure it can reconstruct realistic RGB textures from the DINOV3 latent space.
52
+ 2. **End-to-End Refinement**: The full pipeline is then trained to predict reflection masks from **H**, and fill them using the **<font color="#38761D">Token Inpainter</font>**, ensuring the final output is both visually consistent and physically accurate. The decoder is also fine-tuned at this stage
53
 
54
 
55
 
 
103
  If you use UnReflectAnything in your research or pipeline, please cite our paper:
104
 
105
  ```bibtex
106
+ @InProceedings{Rota_2026_CVPR,
107
+ author = {Rota, Alberto and Kiray, Mert and Karaoglu, Mert Asim and Ruhkamp, Patrick and De Momi, Elena and Navab, Nassir and Busam, Benjamin},
108
+ title = {UnReflectAnything: RGB-Only Highlight Removal by Rendering Synthetic Specular Supervision},
109
+ booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
110
+ month = {June},
111
+ year = {2026},
112
+ pages = {241-250}
 
113
  }
114
 
115
  ```