File size: 2,827 Bytes
4aa5f73 980520d 4aa5f73 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 |
---
license: apache-2.0
tags:
- text-to-image
- image-generation
---
<div align="center">
<h1>ReNeg: Learning Negative Embedding with Reward Guidance</h1>
</div>
<div align="center">
[](https://arxiv.org/abs/2412.19637)
[](https://github.com/AMD-AIG-AIMA/ReNeg)
</div>
We present **ReNeg**, a **Re**ward-guided approach that directly learns **Neg**ative embeddings through gradient descent. The global negative embeddings learned using **ReNeg** exhibit strong generalization capabilities and can be seamlessly adaptable to text-to-image and even text-to-video models. Strikingly simple yet highly effective, **ReNeg** amplifies the visual appeal of outputs from base Stable Diffusion models.
## Examples
Using the [🤗's Diffusers library](https://github.com/huggingface/diffusers) to run ReNeg in a simple and efficient manner.
```bash
pip install diffusers transformers accelerate
git clone https://github.com/AMD-AIG-AIMA/ReNeg.git
```
We provide three negative embeddings, including SD1.4, SD1.5, and SD2.1-base. Running ReNeg with a specific SD version as follows:
```python
import os
from pathlib import Path
import torch
from diffusers import (
StableDiffusionPipeline,
DDIMScheduler,
)
from safetensors.torch import load_file
model_path = "stable-diffusion-v1-5"
neg_embeddings_path = "checkpoints/sd1.5_reneg_emb.safetensors"
pipe = StableDiffusionPipeline.from_pretrained(
model_path,
safety_checker=None,
)
pipe.scheduler = DDIMScheduler.from_pretrained(
model_path, subfolder="scheduler"
)
device = "cuda"
pipe.to(device)
neg_embeddings = load_file(neg_embeddings_path)["embedding"].to(device) # Assuming the key is "embedding"
output = pipe(
"A girl in a school uniform playing an electric guitar.",
negative_prompt_embeds=neg_embeddings,
)
image = output.images[0]
# TextToImageModel is the model you want to evaluate
image.save("output.png")
```
To compare with the inference results using `neg_emb`, you can perform inference using only positive prompt.
+ To perform **inference using only the pos_prompt**, you need to run `inference.py` with `args.prompt_type = only_pos`.
```bash
python inference.py --model_path "your_sd1.5_path" --prompt_type "only_pos" --prompt "A girl in a school uniform playing an electric guitar."
```
## Citation
```
@misc{li2024reneg,
title={ReNeg: Learning Negative Embedding with Reward Guidance},
author={Xiaomin Li, Yixuan Liu, Takashi Isobe, Xu Jia, Qinpeng Cui, Dong Zhou, Dong Li, You He, Huchuan Lu, Zhongdao Wang, Emad Barsoum},
year={2024},
eprint={2412.19637},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
```
|