Duplicate from DiffSynth-Studio/Qwen-Image-Layered-Control

Co-authored-by: kelseye.xh <kelseye@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,54 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/image_1_0_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_1_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_2_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_3_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_4_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_5_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_6_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_7_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_1_input.png filter=lfs diff=lfs merge=lfs -text
+assets/image_2_0_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_2_1_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_2_2_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_2_3_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_2_input.png filter=lfs diff=lfs merge=lfs -text
+assets/image_3_0_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_3_1_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_3_2_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_3_3_0.png filter=lfs diff=lfs merge=lfs -text
+assets/image_3_input.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,136 @@
+---
+license: apache-2.0
+---
+# Qwen-Image-Layered
+
+## Model Introduction
+
+This model is trained based on the model [Qwen/Qwen-Image-Layered](https://modelscope.cn/models/Qwen/Qwen-Image-Layered) using the dataset [artplus/PrismLayersPro](https://modelscope.cn/datasets/artplus/PrismLayersPro), enabling text-controlled extraction of segmented layers.
+
+For more details about training strategies and implementation, feel free to check our [technical blog](https://modelscope.cn/learn/4938).
+
+## Usage Tips
+
+* The model architecture has been changed from multi-image output to single-image output, producing only the layer relevant to the provided text description.
+* The model was trained exclusively on English text, but retains Chinese language understanding capabilities inherited from the base model.
+* The native training resolution is 1024x1024; however, inference at other resolutions is supported.
+* The model struggles to separate multiple entities that are heavily occluded or overlapping, such as the cartoon skeleton head and hat in the examples.
+* The model excels at decomposing poster-like graphics but performs poorly on photographic images, especially those involving complex lighting and shadows.
+* The model supports negative prompts—users can specify content they wish to exclude via negative prompt descriptions.
+
+## Demo Examples
+
+**Some images contain white text on light backgrounds. ModelScope users should click the "☀︎" icon in the top-right corner to switch to dark mode for better visibility.**
+
+### Example 1
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|Input Image|
+|-|
+|![](./assets/image_1_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|Prompt|Output Image|Prompt|Output Image|
+|-|-|-|-|
+|A solid, uniform color with no distinguishable features or objects|![](./assets/image_1_0_0.png)|Text 'TRICK'|![](./assets/image_1_4_0.png)|
+|Cloud|![](./assets/image_1_1_0.png)|Text 'TRICK OR TREAT'|![](./assets/image_1_3_0.png)|
+|A cartoon skeleton character wearing a purple hat and holding a gift box|![](./assets/image_1_2_0.png)|Text 'TRICK OR'|![](./assets/image_1_7_0.png)|
+|A purple hat and a head|![](./assets/image_1_5_0.png)|A gift box|![](./assets/image_1_6_0.png)|
+
+</div>
+
+</div>
+
+### Example 2
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|Input Image|
+|-|
+|![](./assets/image_2_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|Prompt|Output Image|Prompt|Output Image|
+|-|-|-|-|
+|Blue sky, white clouds, a garden with colorful flowers|![](./assets/image_2_0_0.png)|Colorful, intricate floral wreath|![](./assets/image_2_2_0.png)|
+|Girl, wreath, kitten|![](./assets/image_2_1_0.png)|Girl, kitten|![](./assets/image_2_3_0.png)|
+
+</div>
+
+</div>
+
+### Example 3
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|Input Image|
+|-|
+|![](./assets/image_3_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|Prompt|Output Image|Prompt|Output Image|
+|-|-|-|-|
+|A clear blue sky and a turbulent sea|![](./assets/image_3_0_0.png)|Text "The Life I Long For"|![](./assets/image_3_2_0.png)|
+|A seagull|![](./assets/image_3_1_0.png)|Text "Life"|![](./assets/image_3_3_0.png)|
+
+</div>
+
+</div>
+
+## Inference Code
+
+Install DiffSynth-Studio:
+
+```
+git clone https://github.com/modelscope/DiffSynth-Studio.git  
+cd DiffSynth-Studio
+pip install -e .
+```
+
+Model inference:
+
+```python
+from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
+from PIL import Image
+import torch, requests
+
+pipe = QwenImagePipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-Layered-Control", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image-Layered", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
+    ],
+    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
+)
+prompt = "A cartoon skeleton character wearing a purple hat and holding a gift box"
+input_image = requests.get("https://modelscope.oss-cn-beijing.aliyuncs.com/resource/images/trick_or_treat.png", stream=True).raw
+input_image = Image.open(input_image).convert("RGBA").resize((1024, 1024))
+input_image.save("image_input.png")
+images = pipe(
+    prompt,
+    seed=0,
+    num_inference_steps=30, cfg_scale=4,
+    height=1024, width=1024,
+    layer_input_image=input_image,
+    layer_num=0,
+)
+images[0].save("image.png")
+```

README_from_modelscope.md

@@ -0,0 +1,144 @@
+---
+frameworks: PyTorch
+license: Apache License 2.0
+tags: []
+tasks:
+  - text-to-image-synthesis
+base_model:
+  - Qwen/Qwen-Image-Layered
+base_model_relation: finetune
+---
+# Qwen-Image-Layered 
+
+## 模型介绍
+
+本模型基于模型 [Qwen/Qwen-Image-Layered](https://modelscope.cn/models/Qwen/Qwen-Image-Layered) 在数据集 [artplus/PrismLayersPro](https://modelscope.cn/datasets/artplus/PrismLayersPro) 上进行了训练，可以通过文本控制拆分的图层内容。
+
+
+更多关于训练策略和实现细节，欢迎查看我们的[技术博客](https://modelscope.cn/learn/4938)。
+
+## 使用技巧
+
+* 模型结构从多图输出改为了单图输出，仅输出与文本描述相关的图层
+* 模型只用英文文本训练过，但仍从基础模型继承了中文理解能力
+* 模型训练的原生分辨率是1024x1024，支持以其他分辨率进行推理
+* 模型难以拆分“互相遮挡”的多个实体，例如样例中的卡通骷髅头和帽子
+* 模型擅长拆分海报图层，不擅长拆分摄影图像，尤其是存在光影的照片
+* 模型支持负向提示词，可以通过负向提示词描述不希望出现在结果的内容
+
+## 效果展示
+
+**部分图片为纯白色文本，魔搭社区用户请点击页面右上角的“☀︎”切换到暗色模式**
+
+### 样例1
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|输入图|
+|-|
+|![](./assets/image_1_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|提示词|输出图|提示词|输出图|
+|-|-|-|-|
+|A solid, uniform color with no distinguishable features or objects|![](./assets/image_1_0_0.png)|Text 'TRICK'|![](./assets/image_1_4_0.png)|
+|Cloud|![](./assets/image_1_1_0.png)|Text 'TRICK OR TREAT'|![](./assets/image_1_3_0.png)|
+|A cartoon skeleton character wearing a purple hat and holding a gift box|![](./assets/image_1_2_0.png)|Text 'TRICK OR'|![](./assets/image_1_7_0.png)|
+|A purple hat and a head|![](./assets/image_1_5_0.png)|A gift box|![](./assets/image_1_6_0.png)|
+
+</div>
+
+</div>
+
+### 样例2
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|输入图|
+|-|
+|![](./assets/image_2_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|提示词|输出图|提示词|输出图|
+|-|-|-|-|
+|蓝天，白云，一片花园，花园里有五颜六色的花|![](./assets/image_2_0_0.png)|五彩的精致花环|![](./assets/image_2_2_0.png)|
+|少女、花环、小猫|![](./assets/image_2_1_0.png)|少女、小猫|![](./assets/image_2_3_0.png)|
+
+</div>
+
+</div>
+
+### 样例3
+
+<div style="display: flex; justify-content: space-between;">
+
+<div style="width: 30%;">
+
+|输入图|
+|-|
+|![](./assets/image_3_input.png)|
+
+</div>
+
+<div style="width: 66%;">
+
+|提示词|输出图|提示词|输出图|
+|-|-|-|-|
+|一片湛蓝的天空和波涛汹涌的大海|![](./assets/image_3_0_0.png)|文字“向往的生活”|![](./assets/image_3_2_0.png)|
+|一只海鸥|![](./assets/image_3_1_0.png)|文字“生活”|![](./assets/image_3_3_0.png)|
+
+</div>
+
+</div>
+
+## 推理代码
+
+安装 DiffSynth-Studio：
+
+```
+git clone https://github.com/modelscope/DiffSynth-Studio.git  
+cd DiffSynth-Studio
+pip install -e .
+```
+
+模型推理：
+
+```python
+from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
+from PIL import Image
+import torch, requests
+
+pipe = QwenImagePipeline.from_pretrained(
+    torch_dtype=torch.bfloat16,
+    device="cuda",
+    model_configs=[
+        ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-Layered-Control", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
+        ModelConfig(model_id="Qwen/Qwen-Image-Layered", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
+    ],
+    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
+)
+prompt = "A cartoon skeleton character wearing a purple hat and holding a gift box"
+input_image = requests.get("https://modelscope.oss-cn-beijing.aliyuncs.com/resource/images/trick_or_treat.png", stream=True).raw
+input_image = Image.open(input_image).convert("RGBA").resize((1024, 1024))
+input_image.save("image_input.png")
+images = pipe(
+    prompt,
+    seed=0,
+    num_inference_steps=30, cfg_scale=4,
+    height=1024, width=1024,
+    layer_input_image=input_image,
+    layer_num=0,
+)
+images[0].save("image.png")
+```

configuration.json

@@ -0,0 +1 @@
+{"framework":"Pytorch","task":"text-to-image-synthesis"}

qwen_image_layered_control_bf16.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63b1966f0423bdc94d87273b8958de91e0a8f642c635f9113632d09cae3aa4ad
+size 40861043888

transformer/config.json

@@ -0,0 +1,20 @@
+{
+  "_class_name": "QwenImageTransformer2DModel",
+  "_diffusers_version": "0.36.0.dev0",
+  "use_additional_t_cond": true,
+  "attention_head_dim": 128,
+  "axes_dims_rope": [
+    16,
+    56,
+    56
+  ],
+  "guidance_embeds": false,
+  "in_channels": 64,
+  "joint_attention_dim": 3584,
+  "num_attention_heads": 24,
+  "num_layers": 60,
+  "out_channels": 16,
+  "patch_size": 2,
+  "use_layer3d_rope": true,
+  "zero_cond_t": false
+}

transformer/diffusion_pytorch_model-00001-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5353f1dbff8445840012bd2aff2fd209034aa42d0ce623a55f3f542036244a2
+size 9973590960

transformer/diffusion_pytorch_model-00002-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:957d266a7ccdcc9d3f225c82b0afa831ba5084c851b86934b9e4e9f10163b985
+size 9987326040

transformer/diffusion_pytorch_model-00003-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1f0e2bec2869de66f02b53bda77bc11618aba229453be56170209a654ddff0c0
+size 9987307408

transformer/diffusion_pytorch_model-00004-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5244cf56dd45667fc8f373d43550bc187909bc48489f380fa3dcbb02901e7dcf
+size 9930685680

transformer/diffusion_pytorch_model-00005-of-00005.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:45ecb944aad539ceaae9e3ba99dc9f2d650ba034cf4b305b0e83ebce0bb7b55c
+size 982130448