Update README.md
Browse files
README.md
CHANGED
|
@@ -26,19 +26,19 @@ This is a custom-trained **ControlNet** model designed to perform **automatic co
|
|
| 26 |
|
| 27 |
The model takes in a **black-and-white anime styled pictures** (converted to RGB) as conditioning input and generates a **colorized version** using Stable Diffusion.
|
| 28 |
|
| 29 |
-
It is trained to act as a ControlNet module and requires a compatible
|
| 30 |
|
| 31 |
---
|
| 32 |
|
| 33 |
## Training details
|
| 34 |
|
| 35 |
- **Base model:** `John6666/nsfw-anime-xl-v1-sdxl`
|
| 36 |
-
- **Dataset:** Custom dataset of ~6,000 image pairs from **Danbooru-based manga scans**, manually cleaned and resized to `
|
| 37 |
- **Inputs:**
|
| 38 |
- `conditioning_image`: black-and-white manga scan (RGB)
|
| 39 |
-
- `text prompt`: optional (e.g. "
|
| 40 |
-
- **Loss:** MSE with FP16, trained on 1×RTX3090,
|
| 41 |
-
- **Resolution:**
|
| 42 |
- **Scheduler:** default diffusers setup
|
| 43 |
- **Optimizer:** LR: `1.4e-4`
|
| 44 |
|
|
@@ -76,7 +76,7 @@ image.save("colorized.png")
|
|
| 76 |
- Place `diffusion_pytorch_model.safetensors` into your `ComfyUI/models/controlnet/` folder
|
| 77 |
- Make sure to also include the `config.json`
|
| 78 |
- Select this ControlNet in your workflow
|
| 79 |
-
- Use grayscale images
|
| 80 |
|
| 81 |
---
|
| 82 |
|
|
@@ -85,25 +85,25 @@ image.save("colorized.png")
|
|
| 85 |
This version was trained using the SDXL-compatible ControlNet pipeline with the following CLI command:
|
| 86 |
|
| 87 |
```bash
|
| 88 |
-
accelerate launch
|
| 89 |
-
--pretrained_model_name_or_path
|
| 90 |
-
--dataset_name
|
| 91 |
-
--image_column
|
| 92 |
-
--conditioning_image_column
|
| 93 |
-
--caption_column
|
| 94 |
-
--output_dir
|
| 95 |
-
--resolution
|
| 96 |
-
--train_batch_size
|
| 97 |
-
--gradient_accumulation_steps
|
| 98 |
-
--learning_rate
|
| 99 |
-
--num_train_epochs
|
| 100 |
-
--mixed_precision
|
| 101 |
--gradient_checkpointing \
|
| 102 |
-
--checkpointing_steps
|
| 103 |
-
--validation_steps
|
| 104 |
-
--report_to
|
| 105 |
-
--tracker_project_name
|
| 106 |
-
--seed
|
| 107 |
```
|
| 108 |
|
| 109 |
---
|
|
|
|
| 26 |
|
| 27 |
The model takes in a **black-and-white anime styled pictures** (converted to RGB) as conditioning input and generates a **colorized version** using Stable Diffusion.
|
| 28 |
|
| 29 |
+
It is trained to act as a ControlNet module and requires a compatible SDXL base model — such as `nsfw-anime-xl-v1-sdxl` or other anime/manga-focused SDXL models.
|
| 30 |
|
| 31 |
---
|
| 32 |
|
| 33 |
## Training details
|
| 34 |
|
| 35 |
- **Base model:** `John6666/nsfw-anime-xl-v1-sdxl`
|
| 36 |
+
- **Dataset:** Custom dataset of ~6,000 image pairs from **Danbooru-based manga scans**, manually cleaned and resized to `768x768`
|
| 37 |
- **Inputs:**
|
| 38 |
- `conditioning_image`: black-and-white manga scan (RGB)
|
| 39 |
+
- `text prompt`: optional (e.g. "1girl, blue_eyes, blue_hair etc.")
|
| 40 |
+
- **Loss:** MSE with FP16, trained on 1×RTX3090, 12 epochs
|
| 41 |
+
- **Resolution:** 768x768
|
| 42 |
- **Scheduler:** default diffusers setup
|
| 43 |
- **Optimizer:** LR: `1.4e-4`
|
| 44 |
|
|
|
|
| 76 |
- Place `diffusion_pytorch_model.safetensors` into your `ComfyUI/models/controlnet/` folder
|
| 77 |
- Make sure to also include the `config.json`
|
| 78 |
- Select this ControlNet in your workflow
|
| 79 |
+
- Use grayscale images as conditioning inputs
|
| 80 |
|
| 81 |
---
|
| 82 |
|
|
|
|
| 85 |
This version was trained using the SDXL-compatible ControlNet pipeline with the following CLI command:
|
| 86 |
|
| 87 |
```bash
|
| 88 |
+
accelerate launch train_controlnet.py \
|
| 89 |
+
--pretrained_model_name_or_path="John6666/nsfw-anime-xl-v1-sdxl" \
|
| 90 |
+
--dataset_name="SubMaroon/danbooru-colored" \
|
| 91 |
+
--image_column="image" \
|
| 92 |
+
--conditioning_image_column="conditioning_image" \
|
| 93 |
+
--caption_column="prompt" \
|
| 94 |
+
--output_dir="./controlnet-colorization" \
|
| 95 |
+
--resolution=768 \
|
| 96 |
+
--train_batch_size=4 \
|
| 97 |
+
--gradient_accumulation_steps=4 \
|
| 98 |
+
--learning_rate=1.4e-4 \
|
| 99 |
+
--num_train_epochs=12 \
|
| 100 |
+
--mixed_precision="fp16" \
|
| 101 |
--gradient_checkpointing \
|
| 102 |
+
--checkpointing_steps=1000 \
|
| 103 |
+
--validation_steps=1000 \
|
| 104 |
+
--report_to="tensorboard" \
|
| 105 |
+
--tracker_project_name="controlnet-colorization" \
|
| 106 |
+
--seed=42
|
| 107 |
```
|
| 108 |
|
| 109 |
---
|