Instructions to use dhruv1672/data266-pokemon-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use dhruv1672/data266-pokemon-lora with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.bfloat16, device_map="cuda") pipe.load_lora_weights("dhruv1672/data266-pokemon-lora") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
import torch
from diffusers import DiffusionPipeline
# switch to "mps" for apple devices
pipe = DiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.bfloat16, device_map="cuda")
pipe.load_lora_weights("dhruv1672/data266-pokemon-lora")
prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]Pokémon-style LoRA for Stable Diffusion 1.5 (v2.1)
Coursework for DATA 266 Lab 2, Part 2 (Dhruv Patel, Team 28). LoRA adapters for the attention layers of the Stable Diffusion 1.5 UNet, trained on Pokémon artwork with BLIP captions.
Versions
| version | note |
|---|---|
| v1.0 | First experiment: ranks 4 and 16, 1,500 steps, constant learning rate, no EMA. Adapters kept at dhruv1672/data266-pokemon-lora-r4 and dhruv1672/data266-pokemon-lora-r16. |
| v2.0 | Same sweep, but blacked-out safety-checker images were scored as if they were samples, which pushed high ranks down the ranking. Superseded by v2.1. |
| v2.1 | Sweep over rank [4, 8, 16, 32, 64, 128] and alpha, 4000 steps, cosine decay, EMA, min-SNR. Final run: lora_r64_a64. The model card on main was later updated with the blinded human evaluation (Nov 17); the weights are unchanged and the v2.1 tag still points at the release commit. |
Use
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", torch_dtype=torch.float16).to("cuda")
pipe.load_lora_weights("dhruv1672/data266-pokemon-lora", revision="v2.1") # the final run, lora_r64_a64
# any other run of the sweep: pipe.load_lora_weights("dhruv1672/data266-pokemon-lora", revision="v2.1", subfolder="runs/<run name>")
image = pipe("a drawing of a blue and yellow pokemon", num_inference_steps=30, guidance_scale=7.5).images[0]
Data
reach-vb/pokemon-blip-captions at revision 3790f29981f7: 833 images after cleaning, split into 685 train, 64 validation and 84 test images (seed 266). Recoloured copies of the same artwork are kept on one side of the split.
Training
Base stable-diffusion-v1-5/stable-diffusion-v1-5 at revision 451f4fe16113, 512 px. Every run: 4000 steps, batch 4 x 1, AdamW, learning rate 0.0001 with cosine decay after 200 warm-up steps, EMA decay 0.999, min-SNR gamma 5.0, caption dropout 0.1, horizontal flips, mixed precision. The published weights are the EMA weights. The exact config is experiment.yaml.
Sweep (validation images only)
| run | rank | alpha | trainable_parameters | gpu_hours | peak_gpu_gib | val_kid | val_clip | val_flagged | finalist |
|---|---|---|---|---|---|---|---|---|---|
| lora_r128_a128 | 128 | 128 | 25509888 | 0.2221 | 7.443 | 0.01655 | 31.856 | 19 | True |
| lora_r64_a64 | 64 | 64 | 12754944 | 0.2223 | 7.15 | 0.01805 | 31.733 | 13 | True |
| lora_r4_a4 | 4 | 4 | 797184 | 0.2262 | 6.874 | 0.02197 | 32.841 | 9 | False |
| lora_r128_a64 | 128 | 64 | 25509888 | 0.2333 | 7.443 | 0.02242 | 27.641 | 42 | False |
| lora_r8_a8 | 8 | 8 | 1594368 | 0.2717 | 6.892 | 0.0249 | 32.81 | 11 | False |
| lora_r32_a32 | 32 | 32 | 6377472 | 0.2201 | 7.001 | 0.02517 | 32.349 | 13 | False |
| lora_r16_a16 | 16 | 16 | 3188736 | 0.2305 | 6.928 | 0.02523 | 32.484 | 11 | False |
| lora_r128_a256 | 128 | 256 | 25509888 | 0.234 | 7.443 | 0.03134 | 32.999 | 6 | False |
Test results (20 fixed prompts, seeds 0 and 1, DDIM 30 steps, guidance 7.5)
| model | kid | kid_std | clip_score | lpips_diversity | nsfw_flags | scored_images |
|---|---|---|---|---|---|---|
| base | 0.06483 | 0.00832 | 33.22 | 0.8412 | 0 | 40 |
| lora_r128_a128 | 0.02356 | 0.0059 | 32.024 | 0.7496 | 5 | 35 |
| lora_r64_a64 | 0.02031 | 0.00574 | 31.686 | 0.7487 | 6 | 34 |
Final run: lora_r64_a64 (of the two finalists, the lower KID on the test set; ties broken by higher CLIP score). KID uses at most 20 generated images per model, so it is noisy. Images blacked out by the safety checker are counted in nsfw_flags and left out of every metric. Blinded human evaluation: see below.
Human evaluation (blinded)
25 image pairs, each the final model (lora_r64_a64) against the base model for the same prompt and seed, with left/right order randomized. Three raters judged every pair: both team members and one person outside the team (kept anonymous). For each pair they picked the better image for domain style (looks like Pokémon artwork) and for prompt match.
| rater | final model preferred: domain style | final model preferred: prompt match |
|---|---|---|
| Dhruv | 22 / 25 | 14 / 25 |
| Rishitha | 18 / 25 | 18 / 25 |
| Rater 3 (outside the team) | 19 / 25 | 14 / 25 |
| All raters | 59 / 75 (78.7%) | 46 / 75 (61.3%) |
The 25 pairs were drawn with seed 266 from the 34 pairs whose final-model image was not blacked out by the safety checker (6 of the 40 final-model test images were flagged and left out before sampling). The three raters judged the same 25 pairs, so the 75 judgments are not 75 independent pairs. The rating files and the pair key are in the team repository under Part2/Dhruv Patel/results/human_evaluation/.
Limitations
Captions are machine-written and short, so the adapter learns the drawing style better than precise prompt control.
Training images show one creature on a white background; scenes, several characters and out-of-domain prompts are pulled towards that look.
Memory and time were measured on NVIDIA A100-SXM4-40GB with PyTorch limited to 15 GiB; the largest peak of any run was 7.44 GiB, within a 16 GB GPU. See the sweep table for each run.
The final model wins clearly on style but less on prompt match (61.3%), consistent with the short machine-written captions.
License and safety
The adapter inherits CreativeML OpenRAIL-M from the base model. The dataset card declares CC-BY-NC-SA-4.0 and the characters are the property of their owners, so this model is for non-commercial coursework only. Images were generated with the Stable Diffusion safety checker on. A nearest-neighbour check against the training images is in the notebook.
- Downloads last month
- 75
Model tree for dhruv1672/data266-pokemon-lora
Base model
stable-diffusion-v1-5/stable-diffusion-v1-5