GradSPO: A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models

This repository provides public LoRA checkpoints trained with GradSPO for Stable Diffusion v1.5 and SDXL.

GradSPO reframes stepwise preference optimization (SPO) as learning from noisy reward signals, explicitly reducing this noise through gradient guidance. This results in stronger reward signals and achieves improved preference alignment.

All released checkpoints are LoRA weights only and must be loaded on top of their corresponding base models.

The official training code is available at:
https://github.com/JoshuaTTJ/GradSPO

---

Usage

SDXL (LoRA)

from diffusers import StableDiffusionXLPipeline
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)

pipe.load_lora_weights("./sd1_5")

pipe = pipe.to("cuda")

prompt = "A cat holding a sign that says hello world"

generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
    prompt=prompt,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=generator,
    output_type="pil",
).images[0]

image.save("img_sdxl.png")

---

Stable Diffusion v1.5 (LoRA)

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "sd-legacy/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
)

pipe.load_lora_weights("./sdxl")

pipe = pipe.to("cuda")

prompt = "a photo of a cat"

generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
    prompt=prompt,
    guidance_scale=5.0,
    num_inference_steps=20,
    generator=generator,
    output_type="pil",
).images[0]

image.save("img_sd15.png")

---

Citation

If you find GradSPO useful in your research, please consider citing our work:

@inproceedings{
tee2025a,
title={A Gradient Guidance Perspective on Stepwise Preference Optimization for Diffusion Models},
author={Joshua Tian Jin Tee and Hee Suk Yoon and Abu Hanif Muhammad Syarubany and Eunseop Yoon and Chang D. Yoo},
booktitle={The Thirty-ninth Annual Conference on Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=d6lIOnvOX2}
}