How to use from the
Use from the
Diffusers library
pip install -U diffusers transformers accelerate
import torch
from diffusers import DiffusionPipeline

# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("dhruv1672/data266-pokemon-lora")

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

Pokémon-style LoRA for Stable Diffusion 1.5 (v2.1)

Coursework for DATA 266 Lab 2, Part 2 (Dhruv Patel, Team 28). LoRA adapters for the attention layers of the Stable Diffusion 1.5 UNet, trained on Pokémon artwork with BLIP captions.

Versions

version note
v1.0 First experiment: ranks 4 and 16, 1,500 steps, constant learning rate, no EMA. Adapters kept at dhruv1672/data266-pokemon-lora-r4 and dhruv1672/data266-pokemon-lora-r16.
v2.0 Same sweep, but blacked-out safety-checker images were scored as if they were samples, which pushed high ranks down the ranking. Superseded by v2.1.
v2.1 Sweep over rank [4, 8, 16, 32, 64, 128] and alpha, 4000 steps, cosine decay, EMA, min-SNR. Final run: lora_r64_a64. The model card on main was later updated with the blinded human evaluation (Nov 17); the weights are unchanged and the v2.1 tag still points at the release commit.

Use

from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("dhruv1672/data266-pokemon-lora", revision="v2.1")   # the final run, lora_r64_a64
# any other run of the sweep: pipe.load_lora_weights("dhruv1672/data266-pokemon-lora", revision="v2.1", subfolder="runs/<run name>")
image = pipe("a drawing of a blue and yellow pokemon", num_inference_steps=30, guidance_scale=7.5).images[0]

Data

reach-vb/pokemon-blip-captions at revision 3790f29981f7: 833 images after cleaning, split into 685 train, 64 validation and 84 test images (seed 266). Recoloured copies of the same artwork are kept on one side of the split.

Training

Base stable-diffusion-v1-5/stable-diffusion-v1-5 at revision 451f4fe16113, 512 px. Every run: 4000 steps, batch 4 x 1, AdamW, learning rate 0.0001 with cosine decay after 200 warm-up steps, EMA decay 0.999, min-SNR gamma 5.0, caption dropout 0.1, horizontal flips, mixed precision. The published weights are the EMA weights. The exact config is experiment.yaml.

Sweep (validation images only)

run rank alpha trainable_parameters gpu_hours peak_gpu_gib val_kid val_clip val_flagged finalist
lora_r128_a128 128 128 25509888 0.2221 7.443 0.01655 31.856 19 True
lora_r64_a64 64 64 12754944 0.2223 7.15 0.01805 31.733 13 True
lora_r4_a4 4 4 797184 0.2262 6.874 0.02197 32.841 9 False
lora_r128_a64 128 64 25509888 0.2333 7.443 0.02242 27.641 42 False
lora_r8_a8 8 8 1594368 0.2717 6.892 0.0249 32.81 11 False
lora_r32_a32 32 32 6377472 0.2201 7.001 0.02517 32.349 13 False
lora_r16_a16 16 16 3188736 0.2305 6.928 0.02523 32.484 11 False
lora_r128_a256 128 256 25509888 0.234 7.443 0.03134 32.999 6 False

Test results (20 fixed prompts, seeds 0 and 1, DDIM 30 steps, guidance 7.5)

model kid kid_std clip_score lpips_diversity nsfw_flags scored_images
base 0.06483 0.00832 33.22 0.8412 0 40
lora_r128_a128 0.02356 0.0059 32.024 0.7496 5 35
lora_r64_a64 0.02031 0.00574 31.686 0.7487 6 34

Final run: lora_r64_a64 (of the two finalists, the lower KID on the test set; ties broken by higher CLIP score). KID uses at most 20 generated images per model, so it is noisy. Images blacked out by the safety checker are counted in nsfw_flags and left out of every metric. Blinded human evaluation: see below.

Human evaluation (blinded)

25 image pairs, each the final model (lora_r64_a64) against the base model for the same prompt and seed, with left/right order randomized. Three raters judged every pair: both team members and one person outside the team (kept anonymous). For each pair they picked the better image for domain style (looks like Pokémon artwork) and for prompt match.

rater final model preferred: domain style final model preferred: prompt match
Dhruv 22 / 25 14 / 25
Rishitha 18 / 25 18 / 25
Rater 3 (outside the team) 19 / 25 14 / 25
All raters 59 / 75 (78.7%) 46 / 75 (61.3%)

The 25 pairs were drawn with seed 266 from the 34 pairs whose final-model image was not blacked out by the safety checker (6 of the 40 final-model test images were flagged and left out before sampling). The three raters judged the same 25 pairs, so the 75 judgments are not 75 independent pairs. The rating files and the pair key are in the team repository under Part2/Dhruv Patel/results/human_evaluation/.

Limitations

  • Captions are machine-written and short, so the adapter learns the drawing style better than precise prompt control.

  • Training images show one creature on a white background; scenes, several characters and out-of-domain prompts are pulled towards that look.

  • Memory and time were measured on NVIDIA A100-SXM4-40GB with PyTorch limited to 15 GiB; the largest peak of any run was 7.44 GiB, within a 16 GB GPU. See the sweep table for each run.

  • The final model wins clearly on style but less on prompt match (61.3%), consistent with the short machine-written captions.

License and safety

The adapter inherits CreativeML OpenRAIL-M from the base model. The dataset card declares CC-BY-NC-SA-4.0 and the characters are the property of their owners, so this model is for non-commercial coursework only. Images were generated with the Stable Diffusion safety checker on. A nearest-neighbour check against the training images is in the notebook.

Downloads last month
75
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for dhruv1672/data266-pokemon-lora

Adapter
(817)
this model