Add comprehensive model card for OmniTry

README.md

@@ -1,3 +1,77 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: image-to-image
+library_name: diffusers
+---
+
+# OmniTry: Virtual Try-On Anything without Masks
+
+OmniTry is a unified framework that extends Virtual Try-On (VTON) beyond traditional garments to encompass any wearable objects, such as jewelry and accessories. It operates in a mask-free setting for more practical applications. The framework uses a two-staged pipeline, leveraging large-scale unpaired images for mask-free localization and then fine-tuning with paired images for object appearance consistency.
+
+- 📚 [**Paper: OmniTry: Virtual Try-On Anything without Masks**](https://huggingface.co/papers/2508.13632)
+- 🌐 [**Project Page**](https://omnitry.github.io/)
+- 💻 [**GitHub Repository**](https://github.com/Kunbyte-AI/OmniTry)
+- 🤗 [**Hugging Face Spaces Demo**](https://huggingface.co/spaces/Kunbyte/OmniTry)
+- 📊 [**Hugging Face Benchmark Dataset**](https://huggingface.co/datasets/Kunbyte/OmniTry-Bench)
+
+<p align="center">
+  <img src="https://github.com/Kunbyte-AI/OmniTry/raw/main/assets/teaser.png">
+</p>
+
+## Abstract
+Virtual Try-ON (VTON) is a practical and widely-applied task, for which most of existing works focus on clothes. This paper presents OmniTry, a unified framework that extends VTON beyond garment to encompass any wearable objects, e.g., jewelries and accessories, with mask-free setting for more practical application. When extending to various types of objects, data curation is challenging for obtaining paired images, i.e., the object image and the corresponding try-on result. To tackle this problem, we propose a two-staged pipeline: For the first stage, we leverage large-scale unpaired images, i.e., portraits with any wearable items, to train the model for mask-free localization. Specifically, we repurpose the inpainting model to automatically draw objects in suitable positions given an empty mask. For the second stage, the model is further fine-tuned with paired images to transfer the consistency of object appearance. We observed that the model after the first stage shows quick convergence even with few paired samples. OmniTry is evaluated on a comprehensive benchmark consisting of 12 common classes of wearable objects, with both in-shop and in-the-wild images. Experimental results suggest that OmniTry shows better performance on both object localization and ID-preservation compared with existing methods. The code, model weights, and evaluation benchmark of OmniTry will be made publicly available at this https URL .
+
+## News
+*   **[2025.08.20]** 🎉🎉🎉 We release the model weights, inference demo and evaluation benchmark of OmniTry! To experience our advanced version and other related features, please visit our product website [k-fashionshop](https://marketing.k-fashionshop.com/home) (in Chinese) or [visboom](https://www.visboom.com/) (in English).
+
+## Get Started
+
+**Note**: Currently, OmniTry requires at least **28GB** of VRAM for inference under `torch.bfloat16`. We will continue work to decrease memory requirements.
+
+### Download Checkpoints
+1.  Create the checkpoint directory: `mkdir checkpoints`
+
+2.  Download the [FLUX.1-Fill-dev](https://huggingface.co/black-forest-labs/FLUX.1-Fill-dev) into `checkpoints/FLUX.1-Fill-dev`
+
+3.  Download the [LoRA of OmniTry](https://huggingface.co/Kunbyte/OmniTry/blob/main/omnitry_v1_unified.safetensors) into `checkpoints/omnitry_v1_unified.safetensors`. You can also download the `omnitry_v1_clothes.safetensors` that specifically finetuned on the clothe data only.
+
+### Environment Preparation
+Install the environment with `conda`
+```bash
+conda env create -f environment.yml
+conda activate omnitry
+```
+or `pip`:
+```bash
+pip install -r requirements.txt
+```
+
+(Optional) We recommend to install the [flash-attention](https://github.com/Dao-AILab/flash-attention/tree/main) to accelerate the inference process:
+```bash
+pip install flash-attn==2.6.3
+```
+
+### Usage
+For running the gradio demo:
+```bash
+python gradio_demo.py
+```
+
+To change different versions of checkpoints for OmniTry, replace the `lora_path` in `configs/omnitry_v1_unified.yaml`.
+
+## OmniTry-Bench
+We present a unified evaluation benchmark for OmniTry. Please refer to the [OmniTry-Bench](https://github.com/Kunbyte-AI/OmniTry/blob/main/omnitry_bench/README.MD).
+
+## Acknowledgements
+This project is developed on the [diffusers](https://github.com/huggingface/diffusers) and [FLUX](https://github.com/black-forest-labs/flux). We appreciate the contributors for their awesome works.
+
+## Citation
+If you find this codebase useful for your research, please use the following entry.
+```BibTeX
+@article{feng2025omnitry,
+  title={OmniTry: Virtual Try-On Anything without Masks},
+  author={Feng, Yutong and Zhang, Linlin and Cao, Hengyuan and Chen, Yiming and Feng, Xiaoduan and Cao, Jian and Wu, Yuxiong and Wang, Bin},
+  journal={arXiv preprint arXiv:2508.13632},
+  year={2025}
+}
+```