Instructions to use Adeely93/SAGE with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use Adeely93/SAGE with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("Adeely93/SAGE", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
|
| 2 |
+
# SAGE: Structure-Aware Geometric Regularization (ECCV-26)
|
| 3 |
+
|
| 4 |
+
**Paper:** The Illusion of High Utility in Safety Alignment of Text-to-Image Diffusion Models
|
| 5 |
+
**Venue:** ECCV 2026
|
| 6 |
+
**Authors:** Adeel Yousaf, Soumik Ghosh, James Beetham, Amrit Singh Bedi, Mubarak Shah
|
| 7 |
+
**Institution:** University of Central Florida
|
| 8 |
+
**Project Page:** [https://adeelyousaf.github.io/SAGE_ECCV26_Project_Page/](https://adeelyousaf.github.io/SAGE_ECCV26_Project_Page/)
|
| 9 |
+
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
## Overview
|
| 13 |
+
|
| 14 |
+
We show that existing T2I safety alignment methods create an **illusion of high utility** — they appear to have high-utility preservation under coarse metrics (FID, CLIPScore) but suffer significant drops in fine-grained semantic fidelity (TIFA). We trace this to **semantic collapse** in the text encoder embedding space.
|
| 15 |
+
|
| 16 |
+
**SAGE** is a geometry-aware safety alignment method that preserves embedding spread and local similarity structure during fine-tuning, achieving only a **−1.2% TIFA drop** vs. **−6.2% for DES** while maintaining strong safety (Avg. ASR 1.2%).
|
| 17 |
+
|
| 18 |
---
|
| 19 |
+
|
| 20 |
+
## Model
|
| 21 |
+
|
| 22 |
+
This is the fine-tuned **text encoder** of Stable Diffusion v1.4. The UNet remains frozen.
|
| 23 |
+
|
| 24 |
+
```python
|
| 25 |
+
import torch
|
| 26 |
+
from diffusers import StableDiffusionPipeline
|
| 27 |
+
|
| 28 |
+
pipe = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
|
| 29 |
+
checkpoint = torch.load("SAGE.pt", map_location="cpu")
|
| 30 |
+
pipe.text_encoder.load_state_dict(checkpoint)
|
| 31 |
+
pipe = pipe.to("cuda")
|