README.md

metadata

base_model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
library_name: diffusers
pipeline_tag: image-to-image
tags:
  - stable-diffusion-xl
  - inpainting
  - background-editing
  - diffusers

ReFo

Fine-tuned checkpoint of diffusers/stable-diffusion-xl-1.0-inpainting-0.1 for background editing — replacing a photo's background via text-guided inpainting while preserving the original foreground subject.

Model Description

Merged checkpoint: a LoRA trained on synthetic background-replacement pairs has been fused directly into the UNet weights. The model is ready to use as-is, no separate adapter loading required.

• Base model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
• Task: Text-guided background inpainting
• Fine-tuning method: LoRA (rank 16, attention projection layers), merged into base weights
• Training data: Synthetic pairs composited from DIS5K (foreground/matting) and BG-20k (background pool), with auto-generated captions

How to Use

import torch
from PIL import Image
from diffusers import StableDiffusionXLInpaintPipeline

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "esalahterus/refo", torch_dtype=torch.bfloat16
).to("cuda")

source_image = Image.open("path/to/your_image.jpg").convert("RGB")
mask_image = Image.open("path/to/your_mask.png").convert("L")  # white = area to edit, black = area to keep

result = pipe(
    prompt="a high quality photo background, a quiet beach at sunset, photorealistic, detailed, no people, no text",
    negative_prompt="low quality, blurry foreground, distorted subject, watermark, text",
    image=source_image,
    mask_image=mask_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    strength=1.0,  # important: use exactly 1.0 — values like 0.99 only blend lightly instead of fully regenerating the masked area
    generator=torch.Generator(device="cuda").manual_seed(0),  # optional, for reproducible results
).images[0]

result.save("output.png")

No mask image? You can auto-generate a foreground mask with rembg:

from rembg import remove, new_session

session = new_session("u2net")
fg_mask = remove(source_image, session=session, only_mask=True).convert("L")
mask_image = Image.eval(fg_mask, lambda x: 255 - x)  # invert so white = background

Intended Use

• Replacing the background of product photos, portraits, or other subjects via free-text description.
• Suited for automated workflows (e-commerce, portrait editing) that need prompt-driven background control.

Limitations

• Output quality depends heavily on the accuracy of the provided foreground mask.
• strength must be set to exactly 1.0 for the mask region to be fully regenerated; lower values (e.g. 0.99) result in only a light blend and largely ignore the prompt.
• Trained on synthetic compositing data — may underperform on foregrounds with complex edges (fine hair, transparency, etc.).
• Not yet extensively evaluated outside the training data domain (DIS5K + BG-20k).

License

Follows the license of the base model diffusers/stable-diffusion-xl-1.0-inpainting-0.1.