Duplicate from davidelobba/TEMU-VTOFF

Co-authored-by: Davide <davidelobba@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+teaser.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,68 @@
+---
+license: cc-by-nc-4.0
+base_model:
+- stabilityai/stable-diffusion-3-medium-diffusers
+pipeline_tag: image-to-image
+tags:
+- image-generation
+- image-to-image
+- virtual-try-on
+- virtual-try-off
+- diffusion
+- dit
+- stable-diffusion-3
+- multimodal
+- fashion
+- pytorch
+language: en
+datasets:
+- dresscode
+- viton-hd
+---
+
+<div align="center">
+<h1 align="center">TEMU-VTOFF</h1>
+<h3 align="center">Text-Enhanced MUlti-category Virtual Try-Off</h3>
+</div>
+
+<div align="center">
+<picture>
+<source srcset="/davidelobba/TEMU-VTOFF/resolve/main/teaser.png" media="(prefers-color-scheme: dark)">
+<img src="/davidelobba/TEMU-VTOFF/resolve/main/teaser.png" width="75%" alt="TEMU-VTOFF Teaser">
+</source>
+</picture>
+</div>
+
+<div align="center">
+
+**Inverse Virtual Try-On: Generating Multi-Category Product-Style Images from Clothed Individuals**
+[Davide Lobba](https://scholar.google.com/citations?user=WEMoLPEAAAAJ&hl=en&oi=ao)<sup>1,2,\*</sup>, [Fulvio Sanguigni](https://scholar.google.com/citations?user=tSpzMUEAAAAJ&hl=en)<sup>2,3,\*</sup>, [Bin Ren](https://scholar.google.com/citations?user=Md9maLYAAAAJ&hl=en)<sup>1,2</sup>, [Marcella Cornia](https://scholar.google.com/citations?user=DzgmSJEAAAAJ&hl=en)<sup>3</sup>, [Rita Cucchiara](https://scholar.google.com/citations?user=OM3sZEoAAAAJ&hl=en)<sup>3</sup>, [Nicu Sebe](https://scholar.google.com/citations?user=stFCYOAAAAAJ&hl=en)<sup>1</sup>
+<sup>1</sup>University of Trento, <sup>2</sup>University of Pisa, <sup>3</sup>University of Modena and Reggio Emilia
+<sup>*</sup> Equal contribution
+</div>
+
+<div align="center">
+<a href="https://arxiv.org/abs/2505.21062" style="margin: 0 2px;">
+<img src="https://img.shields.io/badge/Paper-Arxiv_2505.21062-darkred.svg" alt="Paper">
+</a>
+<a href="https://temu-vtoff-page.github.io/" style="margin: 0 2px;">
+<img src='https://img.shields.io/badge/Webpage-Project-silver?style=flat&logo=&logoColor=orange' alt='Project Webpage'>
+</a>
+<a href="https://github.com/davidelobba/TEMU-VTOFF" style="margin: 0 2px;">
+<img src="https://img.shields.io/badge/GitHub-Repo-blue.svg?logo=github" alt="GitHub Repository">
+</a>
+<!-- The Hugging Face model badge will be automatically displayed on the model page -->
+</div>
+
+## 💡 Model Description
+
+**TEMU-VTOFF** is a novel dual-DiT (Diffusion Transformer) architecture designed for the Virtual Try-Off task: generating in-shop images of garments worn by a person. By combining a pretrained feature extractor with a text-enhanced generation module, our method can handle occlusions, multiple garment categories, and ambiguous appearances. It further refines generation fidelity via a feature alignment module based on DINOv2.
+
+This model is based on `stabilityai/stable-diffusion-3-medium-diffusers`. The uploaded weights correspond to the finetuned feature extractor and the VTOFF DiT module.
+
+## ✨ Key Features
+Our contribution can be summarized as follows:
+- **🎯 Multi-Category Try-Off**. We present a unified framework capable of handling multiple garment types (upper-body, lower-body, and full-body clothes) without requiring category-specific pipelines.
+- **🔗 Multimodal Hybrid Attention**. We introduce a novel attention mechanism that integrates garment textual descriptions into the generative process by linking them with person-specific features. This helps the model synthesize occluded or ambiguous garment regions more accurately.
+- **⚡ Garment Aligner Module**. We design a lightweight aligner that conditions generation on clean garment images, replacing conventional denoising objectives. This leads to better alignment consistency on the overall dataset and preserves more precise visual retention.
+- **📊 Extensive experiments**. Experiments on the Dress Code and VITON-HD datasets demonstrate that TEMU-VTOFF outperforms prior methods in both the quality of generated images and alignment with the target garment, highlighting its strong generalization capabilities.

transformer/config.json

@@ -0,0 +1,22 @@
+{
+  "_class_name": "SD3Transformer2DModel",
+  "_diffusers_version": "0.33.0.dev0",
+  "_name_or_path": "stabilityai/stable-diffusion-3-medium-diffusers",
+  "attention_head_dim": 64,
+  "caption_projection_dim": 1536,
+  "encoder_depth": 8,
+  "in_channels": 16,
+  "joint_attention_dim": 4096,
+  "num_attention_heads": 24,
+  "num_layers": 24,
+  "out_channels": 16,
+  "patch_size": 2,
+  "pooled_projection_dim": 2048,
+  "pos_embed_max_size": 192,
+  "probing_method": "conv",
+  "projector_dim": 2048,
+  "sample_size": 128,
+  "z_dims": [
+    768
+  ]
+}

transformer/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d11d615e99e02ecbe64e063eb705b4a38361a547348d10440441bfe891111e09
+size 4254964176

transformer_vton/config.json

@@ -0,0 +1,16 @@
+{
+  "_class_name": "SD3Transformer2DModel",
+  "_diffusers_version": "0.33.0.dev0",
+  "_name_or_path": "stabilityai/stable-diffusion-3-medium-diffusers",
+  "attention_head_dim": 64,
+  "caption_projection_dim": 1536,
+  "in_channels": 33,
+  "joint_attention_dim": 4096,
+  "num_attention_heads": 24,
+  "num_layers": 24,
+  "out_channels": 16,
+  "patch_size": 2,
+  "pooled_projection_dim": 2048,
+  "pos_embed_max_size": 192,
+  "sample_size": 128
+}

transformer_vton/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:108ad85011099576200585917042628a0e2b4ee2af09cfba3723e96a9d9a19a1
+size 3830214056