SDXL LoRA DreamBooth - kishlaykumar1995/blinky-sdxl-dbooth-lora-32-1k

Prompt

A photo of sks cartoon character swimming in a pool

Prompt

A sks cartoon relaxing on the beach with a chest full of coins

Model description

These are kishlaykumar1995/blinky-sdxl-dbooth-lora-32-1k LoRA adaption weights for stabilityai/stable-diffusion-xl-base-1.0.

The weights were trained using DreamBooth.

LoRA for the text encoder was enabled: True.

Special VAE used for training: madebyollin/sdxl-vae-fp16-fix.

Trigger words

You should use a photo of sks cartoon character to trigger the image generation.

Download model

Weights for this model are available in Safetensors format.

Download them in the Files & versions tab.

Intended uses & limitations

How to use

# Load the VAE
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16, use_safetensors=True)

base = DiffusionPipeline.from_pretrained(
        "stabilityai/stable-diffusion-xl-base-1.0",
        torch_dtype=torch.float16,
        vae=vae,
        # tokenizer=tokenizer,
        variant="fp16",
).to("cuda")  # .to("cpu")

# Load the scheduler
base.scheduler = EulerDiscreteScheduler.from_config(base.scheduler.config)

# Load the LoRA
base.load_lora_weights("kishlaykumar1995/blinky-sdxl-dbooth-lora-32-1k")

# Generate an image with 75 inference steps
prompt = "A sks cartoon character driving a luxury car"
negative_prompt = "blurry, broken, distorted"

# Set the LoRA scale
lora_scale = 0.8

# Number of steps
num_inference_steps=75

# Guidance scale
guidance_scale = 18.5

# Load the scheduler
base.scheduler = EulerDiscreteScheduler.from_config(base.scheduler.config)

# Create a generator and set the seed
generator = torch.Generator(device="cuda").manual_seed(16312)

# Generate the output image
output = base(prompt=prompt, num_inference_steps=num_inference_steps, generator=generator,
                      negative_prompt=negative_prompt, guidance_scale=guidance_scale,
                      cross_attention_kwargs={"scale": lora_scale})

Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

Training details

[TODO: describe the data used to train the model]