Enabling Preference-driven Unlearning in Few-step Distilled Text-to-Image Diffusion Models
Abstract
A preference-driven unlearning framework tailored to few-step diffusion dynamics enables direct concept removal in distilled models without costly re-distillation.
Text-to-image diffusion models are increasingly distilled into few-step variants and being deployed to enable fast inference. However, their ability to generate harmful or undesired content poses significant safety risks. Data-driven unlearning methods suppress targeted generations by fine-tuning model weights using specialized unlearning objectives. Crucially, these objectives implicitly rely on multi-step denoising dynamics, an assumption that breaks down for few-step distilled (FSD) models, resulting in ineffective forgetting. Furthermore, performing unlearning on the non-distilled base model and subsequently re-distilling it to obtain an unlearned FSD model incurs substantial computational and time overhead, making it impractical in many settings. Hence, we address this limitation with a preference-driven unlearning framework that revisits Direct Preference Optimization (DPO) for diffusion models. We show that standard DPO and its unlearning derivatives, formulated around noise-prediction error, transfer poorly to FSD models due to their altered generation dynamics. To overcome this, we introduce a modified preference optimization formulation explicitly aligned with the few-step generation properties, enabling direct concept removal in FSD models while preserving few-step efficiency and maintaining strong retention of desirable (non-targeted) capabilities. We evaluate our framework primarily on identity and NSFW (nudity) removal tasks and also extend our method to object-level unlearning. Extensive experiments demonstrate consistent and effective forgetting, and strong retention performance, establishing our method as a practical and principled solution for unlearning in FSD models.
Community
In this work, we study how to make fast image-generation models forget unwanted content while preserving their ability to generate useful, high-quality images. We propose CePU that stands for Consistency-enforced Preference-driven Unlearning.
Standard diffusion models generate images through many refinement steps, while faster, distilled models use only a few. We find that existing preference-based unlearning methods often work poorly on these faster models because their training objectives do not match how the models actually generate images.
To address this, we introduce an approach based on consistency: the idea that a model should predict approximately the same final image from neighboring stages of generation. We use this property to encourage reliable predictions for desired images and weaken them for unwanted content. We also include a retention term to preserve the model’s behavior on content we want to keep.
For example, when removing a celebrity’s identity, our goal is to prevent the model from generating that recognizable person while preserving its ability to create portraits with similar poses, lighting, and backgrounds.
We evaluate CePU on celebrity identity removal, nudity suppression, and object removal. Our experiments show that it generally achieves a better balance between suppressing unwanted content and preserving other capabilities. On SD-Turbo, CePU reduces nudity detections by 99.32% across the evaluated prompt sets, including adversarial prompts.
The main contribution of our work is an unlearning objective tailored to how fast diffusion models operate, enabling practical, targeted content suppression while preserving their speed and useful capabilities.
Get this paper in your agent:
hf papers read 2610.10859 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper