Duplicate from alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,137 @@
+---
+license: other
+license_name: flux-1-dev-non-commercial-license
+license_link: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md
+language:
+- en
+base_model:
+- black-forest-labs/FLUX.1-dev
+pipeline_tag: image-to-image
+tags:
+- ComfyUI
+- Inpainting
+library_name: diffusers
+---
+
+<div style="display: flex; justify-content: center; align-items: center;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama.png" alt="alimama" style="width: 20%; height: auto;">
+</div>
+
+[中文版Readme](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Beta/blob/main/README_ZH.md)
+
+# FLUX.1-dev ControlNet Inpainting - Beta
+
+This repository hosts an improved Inpainting ControlNet checkpoint for the [alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha) model, developed by the AlimamaCreative Team.
+
+## 🔥News!
+
+- **[2024.10.16]** We release a 8-step FLUX.1 distilled lora [alimama-creative/FLUX.1-Turbo-Alpha](https://huggingface.co/alimama-creative/FLUX.1-Turbo-Alpha), which is compatible with our Inpainting Controlnet.
+
+## Key Enhancements
+
+Our latest inpainting model brings significant improvements compared to the previous version:
+
+1. **1024 Resolution Support**: Capable of directly processing and generating 1024x1024 resolution images without additional upscaling steps, providing higher quality and more detailed output results.
+2. **Enhanced Detail Generation**: Fine-tuned to capture and reproduce finer details in inpainted areas.
+3. **Improved Prompt Control**: Offers more precise control over generated content through enhanced prompt interpretation.
+
+## Showcase
+
+The following images were generated using a ComfyUI workflow ([click here to download](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json)) with these settings:
+`control-strength` = 1.0, `control-end-percent` = 1.0, `true_cfg` = 1.0
+
+| Image & Prompt Input                                | Alpha Version                                      | Beta Version                                   |
+|-----------------------------------------------------|----------------------------------------------------|----------------------------------------------------|
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>'Write a few lines of words "alimama creative" on the wooden board'</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"a girl with big beautiful white wing"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"red hair"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>" "</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"Albert Einstein"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  Prompt : <i>"Ravello Outdoor Sectional Sofa Set with Coffee Table"</i>
+</div>
+
+### ComfyUI Usage Guidelines:
+
+Download example ComfyUI workflow [here](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json).
+
+- Using `t5xxl-FP16` and `flux1-dev-fp8` models for 30-step inference @1024px & H20 GPU:
+  - GPU memory usage: 27GB
+  - Inference time: 48 seconds (true_cfg=3.5), 26 seconds (true_cfg=1)
+
+- Different results can be achieved by adjusting the following parameters:
+
+| Parameter | Recommended Range | Effect |
+|-----------|------------------|--------|
+| control-strength | 0.6 - 1.0 | Controls how much influence the ControlNet has on the generation. Higher values result in stronger adherence to the control image. |
+| controlend-percent | 0.35 - 1.0 | Determines at which step in the denoising process the ControlNet influence ends. Lower values allow for more creative freedom in later steps. |
+| true-cfg (Classifier-Free Guidance Scale) | 1.0 or 3.5 | Influences how closely the generation follows the prompt. Higher values increase prompt adherence but may reduce image quality. |
+
+- More comprehensive full-image prompts can lead to better overall results. For example, in addition to describing the area to be repaired, you can also describe the background, atmosphere, and style of the entire image. This approach can make the generated results more harmonious and natural.
+
+## Diffusers Integration
+
+1. Install the required diffusers version:
+```shell
+pip install diffusers==0.30.2
+```
+
+2. Clone this repository:
+````shell
+git clone https://github.com/alimama-creative/FLUX-Controlnet-Inpainting.git
+````
+
+3. Configure `image_path`, `mask_path`, and `prompt` in `main.py`, then execute:
+````shell
+python main.py
+````
+
+## Model Specifications
+
+- Training dataset: 15M images from LAION2B and proprietary sources
+- Optimal inference resolution: 1024x1024
+
+## License
+
+Our model weights are released under the [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) Non-Commercial License.

README_ZH.md

@@ -0,0 +1,116 @@
+<div style="display: flex; justify-content: center; align-items: center;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama.png" alt="alimama" style="width: 20%; height: auto;">
+</div>
+
+# FLUX.1-dev ControlNet 图像修复 - Beta版
+
+本仓库包含了由阿里妈妈创意团队开发的[alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha)模型的改进版图像修复ControlNet权重和推理代码。
+
+## 主要改进
+
+与之前版本相比,我们最新的图像修复模型带来了以下改进:
+
+1. **支持1024分辨率**: 能够直接处理和生成1024x1024分辨率的图像,无需额外的放大步骤,提供更高质量和更详细的输出结果。
+2. **增强细节生成**: 经过微调以捕捉和重现修复区域的更精细细节。
+3. **改进提示词控制**: 通过增强的提示词解释,对生成内容提供更精确的控制。
+
+## 展示示例
+
+以下图像是使用ComfyUI工作流生成的([点击此处下载](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json)),设置如下:
+`control-strength` = 1.0, `control-end-percent` = 1.0, `true_cfg` = 1.0
+
+| 图像 & 提示词输入                                    | Alpha 版本                                         | Beta 版本                                        |
+|-----------------------------------------------------|----------------------------------------------------|----------------------------------------------------|
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/1_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>'在木板上写几行"alimama creative"字样'</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/2_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>"一个女孩,有漂亮的大白翅膀"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/3_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>"红色头发"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/4_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>" "</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/5_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>"阿尔伯特·爱因斯坦"</i>
+</div>
+
+<div align = "center">
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_0.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_1.png" width = "30%" style="display:inline-block;" />
+  <img src="https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/6_2.png" width = "30%" style="display:inline-block;"/>
+<br>
+  提示词 : <i>"Ravello户外沙发组合配咖啡桌"</i>
+</div>
+
+### ComfyUI使用指南:
+
+下载示例ComfyUI工作流[点击这里](https://huggingface.co/alimama-creative/FLUX.1-dev-Controlnet-Inpainting-Alpha/resolve/main/images/alimama-creative-flux-inapint-cn-beta.json)。
+
+- 使用`t5xxl-FP16`和`flux1-dev-fp8`模型进行30步推理@1024px & H20 GPU:
+  - GPU内存使用: 27GB
+  - 推理时间: 48秒 (true_cfg=3.5), 26秒 (true_cfg=1)
+
+- 通过调整以下参数可以获得不同的结果:
+
+| 参数 | 推荐范围 | 效果 |
+|-----------|------------------|--------|
+| control-strength | 0.6 - 1.0 | 控制ControlNet对生成的影响程度。较高的值会使生成结果更严格地遵循控制图像。 |
+| controlend-percent | 0.35 - 1.0 | 决定ControlNet影响在去噪过程中的哪一步结束。较低的值在后期步骤中允许更多的创意自由。 |
+| true-cfg (无分类器引导尺度) | 1.0 或 3.5 | 影响生成结果对提示词的遵循程度。较高的值会增加对提示词的遵循,但可能降低图像质量。 |
+
+- 更全面的整图提示词可以带来更好的整体效果。例如,除了描述需要修复的区域,你还可以描述整个图像的背景、氛围和风格。这种方法可以使生成的结果更加和谐自然。
+
+## Diffusers 推理
+
+1. 安装所需的diffusers版本:
+```shell
+pip install diffusers==0.30.2
+```
+
+2. 克隆此仓库:
+```shell
+git clone https://github.com/alimama-creative/FLUX-Controlnet-Inpainting.git
+```
+
+3. 在`main.py`中配置`image_path`、`mask_path`和`prompt`,然后执行:
+```shell
+python main.py
+```
+
+## 模型说明
+
+- 训练数据集: 来自LAION2B和专有来源的1500万张图像
+- 最佳推理分辨率: 1024x1024
+
+## 许可证
+
+我们的模型权重在[FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md)非商业许可下发布。

config.json

@@ -0,0 +1,20 @@
+{
+  "_class_name": "FluxControlNetModel",
+  "_diffusers_version": "0.30.2",
+  "_name_or_path": "/data/oss_bucket_0/linjinpeng.ljp/exp_flux/r768_bs96_adamw_lr5e-6_bf16_cfg3.5_sin0_dou6_s11/checkpoint-50000",
+  "attention_head_dim": 128,
+  "axes_dims_rope": [
+    16,
+    56,
+    56
+  ],
+  "extra_condition_channels": 4,
+  "guidance_embeds": true,
+  "in_channels": 64,
+  "joint_attention_dim": 4096,
+  "num_attention_heads": 24,
+  "num_layers": 6,
+  "num_single_layers": 0,
+  "patch_size": 1,
+  "pooled_projection_dim": 768
+}

diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca46c5f7b5de02caee7c069f2aedbf628af8def8578319ceae3be1588d448448
+size 4281803800