import torch
from diffusers import DiffusionPipeline
from diffusers.utils import load_image
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("ghoskno/Color-Canny-Controlnet-model", dtype=torch.bfloat16, device_map="cuda")
prompt = "Turn this cat into a dog"
input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
image = pipe(image=input_image, prompt=prompt).images[0]Color-Canny CantrolNet
These are ControlNet checkpoints trained on runwayml/stable-diffusion-v1-5, using fused color and canny edge as conditioning.
You can find some example images in the following.
Examples
Color examples
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the sea
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
prompt: a concept art of by Makoto Shinkai, a girl is standing in the middle of the grass
negative prompt: text, bad anatomy, blurry, (low quality, blurry)
Brightness examples
This model also can be used to control image brightness. The following images are generated with different brightness conditioning image and controlnet strength(0.5 ~ 0.7).
Limitations and Bias
- No strict control by input color
- Sometimes generate image with confusion When color description in prompt
Training
Dataset We train this model on laion-art dataset with 2.6m images, the processed dataset can be found in ghoskno/laion-art-en-colorcanny.
Training Details
Hardware: Google Cloud TPUv4-8 VM
Optimizer: AdamW
Train Batch Size: 4 x 4 = 16
Learning rate: 0.00001 constant
Gradient Accumulation Steps: 4
Resolution: 512
Train Steps: 36000
- Downloads last month
- 29
Model tree for ghoskno/Color-Canny-Controlnet-model
Base model
runwayml/stable-diffusion-v1-5