upload readme

README.md

@@ -1,3 +1,183 @@
----
-license: mit
----
+<div style="display: flex; justify-content: center; align-items: center;">
+  <img src="./images/images_alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;">
+  <img src="./images/images_alimama.png" alt="alimama" style="width: 20%; height: auto;">
+</div>
+
+EcomID aims to generate customized images from a single reference ID image, ensuring strong semantic consistency while being controlled by keypoints.
+
+This repository provides the EcomID method and model, combining the strengths of PuLID and InstantID for better background consistency, facial keypoint control, and realistic facial representation with improved similarity.
+
+# EcomID Overview
+
+## EcomID Structure
+  <img src="./images/overflow.png" alt="alibaba" style="width: 100%; height: auto; margin-right: 5%;">
+
+
+- **IP-Adapter of PuLID**: EcomID incorporates the ID-Encoder and cross-attention components from PuLID, trained with alignment loss. 
+This method effectively reduces the interference of ID embeddings on text embeddings within the cross-attention part, minimizing disruption to the underlying model's text-to-image capabilities.
+- **InstantID’s IdentityNet Architecture**: Utilizing **a dataset of 2 million aesthetically pleasing portrait images**, IdentityNet enhances keypoint control, improving ID consistency and facial realism. During training, the IP-adapter is frozen, and only the IdentityNet is trained. Facial landmarks are used as conditional inputs, while face embeddings are integrated into IdentityNet via cross-attention.
+
+# Show Cases
+## Comparison with Other Methods
+### 1、Preserved Text-to-Image Capability
+
+<table>
+    <tr>
+        <th style="width: 28%;">Prompt</th>
+        <th style="width: 24%;">Reference Image</th>
+        <th style="width: 24%;">EcomID</th>
+        <th style="width: 24%;">InstantID</th>
+    </tr>
+    <tr>
+        <td>girl, white skin, black hair, long wavy hair, <span style="color:red"><strong>in European style living room, Retro tone, decorations</strong></span>, depth of field.</td>
+        <td><img src="images/show_case/50.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/49.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/48.png" alt="InstantID图像" width="100%"></td>
+    </tr>
+<table>
+
+As shown above, EcomID ***preserves background generation abilities while minimizing stylization, greatly enhancing realism***. 
+The visualizations highlight more authentic portraits with improved background semantic consistency, showcasing EcomID's advantage in generating realistic images.
+
+### 2、Improved Facial Control and Consistency
+<table>
+    <tr>
+        <th style="width: 24%;">Prompt</th>
+        <th style="width: 19%;">Reference Image</th>
+        <th style="width: 19%;">EcomID</th>
+        <th style="width: 19%;">InstantID</th>
+        <th style="width: 19%;">PuLID</th>
+    </tr>
+    <tr>
+        <td>A close-up portrait of a man standing in the library, holding <span style="color:red"><strong>two smiling toddlers</strong></span> next to him.</td>
+        <td><img src="images/show_case/20.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/17.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/18.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/19.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+<table>
+
+As shown above, EcomID employs keypoints as conditional inputs for training, ***allowing for precise adjustments of facial positions, sizes, and orientations***. This capability ensures that the generated portraits are more controllable while further enhancing facial similarity and the overall quality of the images.
+
+### More showcases
+EcomID enhances portrait representation, delivering a more authentic and aesthetically pleasing appearance while ensuring semantic consistency and greater internal ID similarity (i.e., traits that do not vary with age, hairstyle, glasses, or other physical changes).
+
+<table>
+    <tr>
+        <th style="width: 24%;">Prompt</th>
+        <th style="width: 19%;">Reference Image</th>
+        <th style="width: 19%;">EcomID</th>
+        <th style="width: 19%;">InstantID</th>
+        <th style="width: 19%;">PuLID</th>
+    </tr>
+    <tr>
+        <td>A close-up portrait of a <span style="color:red"><strong>little girl with double braids</strong></span>, wearing a white dress, standing on the beach during sunset.</td>
+        <td><img src="images/show_case/21.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/22.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/23.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/24.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td>
+        <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>Agrizzled detective, <span style="color:red"><strong>fedora</strong></span> casting a shadow over his square jaw, a <span style="color:red"><strong>cigar dangling from his lips</strong></span>, his trench coat evocative of film noir, in a <span style="color:red"><strong>rainy alley</strong></span>.</td>
+        <td><img src="images/show_case/25.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/26.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/27.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/28.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>A smiling girl with <span style="color:red"><strong>bangs and long hair</strong></span> in a school uniform stands under cherry trees, holding a book.</td>
+        <td><img src="images/show_case/29.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/30.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/31.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/32.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>A <span style="color:red"><strong>very old</strong></span> witch, wearing a black cloak, with a pointed hat, holding a magic wand, against a background of a misty forest.</td>
+        <td><img src="images/show_case/33.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/34.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/35.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/36.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>A man clad in cyberpunk fashion: <span style="color:red"><strong>neon accents, reflective sunglasses，</strong></span> and a leather jacket with glowing circuit patterns. He stands stoically amidst a soaked cityscape.</td>
+        <td><img src="images/show_case/37.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/38.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/39.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/40.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+
+</table>
+
+### More Base Models, Resolutions, and Styles
+<table>
+    <tr>
+        <th style="width: 12%;">SDXL models</th>
+        <th style="width: 24%;">Prompt</th>
+        <th style="width: 16%;">Reference Image</th>
+        <th style="width: 16%;">EcomID</th>
+        <th style="width: 16%;">InstantID</th>
+        <th style="width: 16%;">PuLID</th>
+    </tr>
+    <tr>
+        <td>sd-xl-base-1.0</td>
+        <td>girl, solo, brown hair, holding a little teddy bear on her hands, wearing a school uniform, standing in the library, <span style="color:red"><strong>cartoon style</strong></span>.</td>
+        <td><img src="images/show_case/1.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/2.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/3.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/4.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>EcomXL</td>
+        <td>A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td>
+        <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>DreamShaperXL</td>
+        <td>solo, looking_at_viewer, smile, brown_hair, upper_body, open_clothes, teeth, open_jacket, black_jacket, blurry_background, realistic</td>
+        <td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/6.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/7.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/8.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+    <tr>
+        <td>leosam_xl_v7</td>
+        <td>A close-up portrait of a girl, solo, dress, jewelry, beach and sea, pink_dress, realistic.</td>
+        <td><img src="images/show_case/9.png" alt="参考图像" width="100%"></td>
+        <td><img src="images/show_case/15.png" alt="EcomID图像" width="100%"></td>
+        <td><img src="images/show_case/14.png" alt="InstantID图像" width="100%"></td>
+        <td><img src="images/show_case/16.png" alt="PuLID图像" width="100%"></td>
+    </tr>
+
+</table>
+
+### Notes
+- Unless otherwise specified, the showcases are generated using the base model EcomXL, which is also highly compatible with various other SDXL-based models, such as [leosams-helloworld-xl](https://civitai.com/models/43977/leosams-helloworld-xl), [dreamshaper-xl](https://civitai.com/models/112902/dreamshaper-xl), [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and so on.
+- It works very well with SDXL Turbo/Lighting, [EcomXL Inpainting ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_inpaint) and [EcomXL Softedge ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_softedge).
+
+# How to use
+
+## ComfyUI
+
+- The EcomID_ComfyUI node has been released: [click here](https://code.alibaba-inc.com/ruxue.wrx/EcomID_ComfyUI)
+
+# Training Details
+
+The model is trained on 2M Taobao images, where the proportion of human faces is greater than 3%. The images have a resolution greater than 800, and the aesthetic score is above 5.5.
+
+Mixed precision: fp16
+
+Learning rate: 1e-4
+
+Batch size: 2
+
+Image size: 1024x1024

diffusion_pytorch_model.safetensors

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:0f21e607c9582b1df7a14f6b99597c4a6dda4cb37e4f6675db8985971e4b1cf2
-size 5004167864