File size: 4,095 Bytes
fcb72eb 46b6a5b fcb72eb 59da8a0 46b6a5b d06ca26 46b6a5b 0b3513f 46b6a5b 0b3513f 46b6a5b 4918ff0 46b6a5b db04c44 0b3513f 46b6a5b 59da8a0 1099b93 59da8a0 46b6a5b 9f23e65 4918ff0 46b6a5b 4918ff0 46b6a5b 4918ff0 46b6a5b 1099b93 59da8a0 9f23e65 59da8a0 4918ff0 59da8a0 4918ff0 46b6a5b 9f23e65 0b3513f 9f23e65 0b3513f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 | ---
license: openrail
datasets:
- ChristophSchuhmann/improved_aesthetics_6.5plus
language:
- en
---
Controls image generation by edge maps generated with [EdgeDrawing Parameter-Free](https://github.com/CihanTopal/ED_Lib).
* Based on my GitHub monologs at [Edge Drawing - a Canny alternative](https://github.com/lllyasviel/ControlNet/discussions/318)
* For usage see the model page on [Civitai.com](https://civitai.com/models/149740).
* To generate edpf maps you can use the script [edpf.py](https://gitlab.com/-/snippets/3601881).
* For evaluation see the corresponding .zip with images in "files".
* To run your own evaluations you can use [inference.py](https://gitlab.com/-/snippets/3602096).
**EdgeDrawing Parameter-Free**
**Example**
sampler=UniPC steps=20 cfg=7.5 seed=0 batch=9 model: v1-5-pruned-emaonly.safetensors cherry-picked: 1/9
prompt: _a detailed high-quality professional photo of swedish woman standing in front of a mirror, dark brown hair, white hat with purple feather_
**Canndy Edge Detection (default in Automatic1111)**
# Image dataset
* [laion2B-en aesthetics>=6.5 dataset](https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_6.5plus)
* `--min_image_size 512 --max_aspect_ratio 2 --resize_mode="center_crop" --image_size 512`
* resulting in 180k images
# Training
```
accelerate launch train_controlnet.py ^
--pretrained_model_name_or_path="runwayml/stable-diffusion-v1-5" ^
--output_dir="control-edgedrawing-[version]-fp16/" ^
--dataset_name="mydataset" ^
--mixed_precision="fp16" ^
--resolution=512 ^
--learning_rate=1e-5 ^
--train_batch_size=1 ^
--gradient_accumulation_steps=4 ^
--gradient_checkpointing ^
--use_8bit_adam ^
--enable_xformers_memory_efficient_attention ^
--set_grads_to_none ^
--seed=0
```
# Versions
**Experiment 5 - control-edgedrawing-cv480edpf-drop0+50-fp16-checkpoint-118000**
see experiment 4. resumed with epoch 2 from 90000 using `--proportion_empty_prompts=0.5` => results became worse, CN didn't pick up on no-prompts (I also tried checkpoint-104000). restarting with 50% drop.
**Experiment 4 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-90000**
Conditioning images generated with [edpf.py](https://gitlab.com/-/snippets/3601881) using [opencv-contrib-python::ximgproc::EdgeDrawing](https://docs.opencv.org/4.8.0/d1/d1c/classcv_1_1ximgproc_1_1EdgeDrawing.html).
```
ed = cv2.ximgproc.createEdgeDrawing()
params = cv2.ximgproc.EdgeDrawing.Params()
params.PFmode = True
ed.setParams(params)
edges = ed.detectEdges(image)
edge_map = ed.getEdgeImage(edges)
```
90000 steps (45000 steps on original, 45000 steps with left-right flipped images)
**Experiment 3 - control-edgedrawing-cv480edpf-drop0-fp16-checkpoint-45000**
see experiment 4. 45000 steps. This is version 0.1 on civitai.
**Experiment 2 - control-edgedrawing-default-noisy-drop0-fp16-checkpoint-40000**
Images converted with https://github.com/shaojunluo/EDLinePython
Default settings are:
`smoothed=False`
```
{ 'ksize' : 5
, 'sigma' : 1.0
, 'gradientThreshold': 36
, 'anchorThreshold' : 8
, 'scanIntervals' : 1
}
```
`smoothed=True`, but no empty prompts. Trained for 40000 steps with default settings => conditioning images are too noisy.
**Experiment 1 - control-edgedrawing-default-drop50-fp16-checkpoint-40000**
Same as experiment 2.
Update: bug in algorithm produces too sparse images on default, see https://github.com/shaojunluo/EDLinePython/issues/4
additional arguments: `--proportion_empty_prompts=0.5`. Trained for 40000 steps with default settings => empty prompts were probably too excessive
# Question and answers
**Q: What's the point of another edge control net anyway?**
A: 🤷
|