README.md

---
license: fair-noncommercial-research-license
extra_gated_fields:
  First Name: text
  Last Name: text
  Date of birth: date_picker
  Country: country
  Affiliation: text
  Job title:
    type: select
    options:
    - Student
    - Research Graduate
    - AI researcher
    - AI developer/engineer
    - Reporter
    - Other
  geo: ip_location
  By clicking Submit below I accept the terms of the license and acknowledge that the information I provide will be collected stored processed and shared in accordance with the Meta Privacy Policy: checkbox
extra_gated_description: >-
  The information you provide will be collected, stored, processed and shared in
  accordance with the [Meta Privacy
  Policy](https://www.facebook.com/privacy/policy/).
extra_gated_button_content: Submit

language:
- en
tags:
- meta-ai
- meta-pytorch
---

Please be sure to provide your full legal name, date of birth, and full organization name with all corporate identifiers. Avoid the use of acronyms and special characters. Failure to follow these instructions may prevent you from accessing this model and others on Hugging Face. You will not have the ability to edit this form after submission, so please ensure all information is accurate.

# Model Card for [Meta AI BLenDeR](https://arxiv.org/abs/2601.20246) weights

This repository contains pre-trained LoRA weights and Textual Inversion embeddings for Stable Diffusion 1.5 to generate images of birds.

The weights are used in the paper *BLenDeR: Blended Text Embeddings and Diffusion Residuals for Intra-Class Image Synthesis in Deep Metric Learning* (see the [official code repository](https://github.com/facebookresearch/meta_ai_blender)).

BLenDeR is an inference-time sampling method for personalized diffusion models that combines text embedding interpolation with novel set-theoretic residual set operations (union, intersection, and difference over denoising residuals) to synthesize class-specific concepts with novel attributes.



### How to use

Loading LoRA weights and Textual Inversion embeddings:

```python
from diffusers import DDPMScheduler, StableDiffusionPipeline
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

pipeline = StableDiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    torch_dtype=torch.float16,
)

pipeline.scheduler = DDPMScheduler.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5",
    subfolder="scheduler",
)
pipeline.load_lora_weights("./lora-20000/lora_weights/")

pipeline.to(device)

# Loading Textual Inversion embeddings
data = torch.load(
    "./learned_embeds-steps-20000.bin",
    map_location="cpu",
)

words = sorted(data["learned_embeds_dict"].keys())

num_added_tokens = pipeline.tokenizer.add_tokens(words)

if num_added_tokens != len(words):
    raise ValueError("Some tokens are already in the tokenizer.")

pipeline.text_encoder.resize_token_embeddings(len(pipeline.tokenizer))
token_ids = pipeline.tokenizer.convert_tokens_to_ids(words)

weights = pipeline.text_encoder.get_input_embeddings().weight.data

for i, word in enumerate(words):
    weights[token_ids[i]] = data["learned_embeds_dict"][word]
```

Obtain a Textual Inversion token and generate an image for the class represented by the token:

```python
ti_embeddings = torch.load("learned_embeds-steps-20000.bin")
ti_token_strs = list(ti_embeddings["learned_embeds_dict"].keys())
class_0_token = ti_token_strs[0] # e.g. '<000.Bird_Name>'

prompt = f"a photo of a {class_0_token} bird." # prompt format for image generation

image = pipeline(prompt).images[0]

```

See the [official BLenDeR repository](https://github.com/facebookresearch/meta_ai_blender) for more code examples and the full BLenDeR denoising loop.


## Citing BLenDeR

If you find BLenDeR useful, please consider citing:

```bibtex
@article{kolf2026blender,
  title={BLenDeR: Blended Text Embeddings and Diffusion Residuals for Intra-Class Image Synthesis in Deep Metric Learning},
  author={Kolf, Jan Niklas and Tezcan, Ozan and Theiss, Justin and Kim, Hyung Jun and Bao, Wentao and Bhushanam, Bhargav and Gupta, Khushi and Kejariwal, Arun and Damer, Naser and Boutros, Fadi},
  journal={arXiv preprint arXiv:2601.20246},
  year={2026}
}
```

## License
[FAIR Noncommercial Research License](https://github.com/facebookresearch/pixio?tab=License-1-ov-file#readme)

This release is intended to support the open-source research community and fundamental research. Users are expected to leverage the artifacts for research purposes and make research findings arising from the artifacts publicly available for the benefit of the research community.