README.md

---
base_model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
library_name: diffusers
pipeline_tag: image-to-image
tags:
- stable-diffusion-xl
- inpainting
- background-editing
- diffusers
---

# ReFo

Fine-tuned checkpoint of [`diffusers/stable-diffusion-xl-1.0-inpainting-0.1`](https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1) for **background editing** — replacing a photo's background via text-guided inpainting while preserving the original foreground subject.

## Model Description

**Merged** checkpoint: a LoRA trained on synthetic background-replacement pairs has been fused directly into the UNet weights. The model is ready to use as-is, no separate adapter loading required.

- **Base model:** `diffusers/stable-diffusion-xl-1.0-inpainting-0.1`
- **Task:** Text-guided background inpainting
- **Fine-tuning method:** LoRA (rank 16, attention projection layers), merged into base weights
- **Training data:** Synthetic pairs composited from [DIS5K](https://github.com/xuebinqin/DIS) (foreground/matting) and [BG-20k](https://huggingface.co/datasets/unography/BG-20k) (background pool), with auto-generated captions

## How to Use

```python
import torch
from PIL import Image
from diffusers import StableDiffusionXLInpaintPipeline

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "esalahterus/refo", torch_dtype=torch.bfloat16
).to("cuda")

source_image = Image.open("path/to/your_image.jpg").convert("RGB")
mask_image = Image.open("path/to/your_mask.png").convert("L")  # white = area to edit, black = area to keep

result = pipe(
    prompt="a high quality photo background, a quiet beach at sunset, photorealistic, detailed, no people, no text",
    negative_prompt="low quality, blurry foreground, distorted subject, watermark, text",
    image=source_image,
    mask_image=mask_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    strength=1.0,  # important: use exactly 1.0 — values like 0.99 only blend lightly instead of fully regenerating the masked area
    generator=torch.Generator(device="cuda").manual_seed(0),  # optional, for reproducible results
).images[0]

result.save("output.png")
```

No mask image? You can auto-generate a foreground mask with [`rembg`](https://github.com/danielgatis/rembg):

```python
from rembg import remove, new_session

session = new_session("u2net")
fg_mask = remove(source_image, session=session, only_mask=True).convert("L")
mask_image = Image.eval(fg_mask, lambda x: 255 - x)  # invert so white = background
```

## Intended Use

- Replacing the background of product photos, portraits, or other subjects via free-text description.
- Suited for automated workflows (e-commerce, portrait editing) that need prompt-driven background control.

## Limitations

- Output quality depends heavily on the accuracy of the provided foreground mask.
- `strength` must be set to exactly `1.0` for the mask region to be fully regenerated; lower values (e.g. 0.99) result in only a light blend and largely ignore the prompt.
- Trained on synthetic compositing data — may underperform on foregrounds with complex edges (fine hair, transparency, etc.).
- Not yet extensively evaluated outside the training data domain (DIS5K + BG-20k).

## License

Follows the license of the base model `diffusers/stable-diffusion-xl-1.0-inpainting-0.1`.