File size: 4,271 Bytes
3fa8468 7f00d30 3fa8468 4f75903 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
---
license: mit
language:
- en
library_name: diffusers
tags:
- text-to-image
- personalization
- adapter
- stable-diffusion
- flux
- diffusers
base_model:
- runwayml/stable-diffusion-v1-5
- stabilityai/stable-diffusion-2-1
- stabilityai/stable-diffusion-xl-base-1.0
- stabilityai/stable-diffusion-3.5-large
- black-forest-labs/FLUX.1-dev
pipeline_tag: text-to-image
---
# DrUM (**D**raw **You**r **M**ind)
**DrUM** enables **personalized text-to-image (T2I) generation by integrating reference prompts** into T2I diffusion models. It works with **foundation T2I models such as Stable Diffusion v1/v2/XL/v3 and FLUX**, without requiring additional fine-tuning. DrUM leverages **condition-level modeling in the latent space using a transformer-based adapter**, and integrates seamlessly with **open-source text encoders such as OpenCLIP and Google T5**.
This repository provides the necessary components to run DrUM for **inference**. For the full source code, training scripts, and detailed documentation, please visit our official **[GitHub repository](https://github.com/Burf/DrUM)** and read the **research paper [[iccv](https://openaccess.thecvf.com/content/ICCV2025/papers/Kim_Draw_Your_Mind_Personalized_Generation_via_Condition-Level_Modeling_in_Text-to-Image_ICCV_2025_paper.pdf)] [[supp](https://openaccess.thecvf.com/content/ICCV2025/supplemental/Kim_Draw_Your_Mind_ICCV_2025_supplemental.pdf)] [[arXiv](https://arxiv.org/abs/2508.03481)]**.
<p align="center">
<img src="teaser.png" width="95%">
</p>
## Quickstart
This model is designed for easy use with the `diffusers` library as a custom pipeline.
### Installation
```bash
pip install torch torchvision diffusers transformers accelerate safetensors huggingface-hub
```
### Usage
```python
import torch
from diffusers import DiffusionPipeline
from pipeline import DrUM
# Load pipeline and attach DrUM
#drum = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", custom_pipeline = "Burf/DrUM", pipeline = "runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16, device = "cuda")
pipeline = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype = torch.bfloat16).to("cuda")
drum = DrUM(pipeline)
# Generate personalized images
images = drum(
prompt = "a photograph of an astronaut riding a horse",
ref = ["A retro-futuristic space exploration movie poster with bold, vibrant colors"],
weight = [1.0],
alpha = 0.3
)
images[0].save("personalized_image.png")
```
## Supported foundation T2I models
DrUM works with a wide variety of foundation T2I models that uses text encoders with same weights:
| Architecture | Pipeline | Text encoder | DrUM weight |
|--------------|----------------|-|-------------|
| Stable Diffusion v1 | `runwayml/stable-diffusion-v1-5`, `prompthero/openjourney-v4`,<br>`stablediffusionapi/realistic-vision-v51`,`stablediffusionapi/deliberate-v2`,<br>`stablediffusionapi/anything-v5`, `WarriorMama777/AbyssOrangeMix2`, ... | `openai/clip-vit-large-patch14` | `L.safetensors` |
| Stable Diffusion v2 | `stabilityai/stable-diffusion-2-1`, ... | `openai/clip-vit-huge-patch14` | `H.safetensors` |
| Stable Diffusion XL | `stabilityai/stable-diffusion-xl-base-1.0`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k` | `L.safetensors`,<br>`bigG.safetensors` |
| Stable Diffusion v3 | `stabilityai/stable-diffusion-3.5-large`<br>`stabilityai/stable-diffusion-3.5-medium`, ... | `openai/clip-vit-large-patch14`,<br>`laion/CLIP-ViT-bigG-14-laion2B-39B-b160k`,<br>`google/t5-v1_1-xxl` | `L.safetensors`,<br>`bigG.safetensors`,<br>`T5.safetensors` |
| FLUX | `black-forest-labs/FLUX.1-dev`, ... | `openai/clip-vit-large-patch14`,<br>`google/t5-v1_1-xxl` | `L.safetensors`<br>`T5.safetensors` |
## Citation
```
@InProceedings{kim2025drum,
author = {Kim, Hyungjin and Ahn, Seokho and Seo, Young-Duk},
title = {Draw Your Mind: Personalized Generation via Condition-Level Modeling in Text-to-Image Diffusion Models},
booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
month = {October},
year = {2025},
pages = {17171-17180}
}
```
## License
This project is licensed under the MIT License. |