Image-to-Image
Diffusers
Safetensors
Core ML
StableDiffusionInpaintPipeline
clover-image
inpainting
stable-diffusion
Instructions to use neonforestmist/Clover-Image-Tiny-Inpaint with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use neonforestmist/Clover-Image-Tiny-Inpaint with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("neonforestmist/Clover-Image-Tiny-Inpaint", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Add validated inpainting example and stable sampler guidance
Browse files- .gitattributes +2 -0
- README.md +27 -2
- examples/mask-doorway.png +0 -0
- examples/result-cat.png +3 -0
- examples/source-greenhouse.png +3 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
examples/result-cat.png filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
examples/source-greenhouse.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -30,18 +30,20 @@ the VAE encoder needed to prepare the masked-image latent on device.
|
|
| 30 |
## Diffusers example
|
| 31 |
|
| 32 |
```python
|
| 33 |
-
from diffusers import AutoPipelineForInpainting
|
| 34 |
from diffusers.utils import load_image
|
| 35 |
|
| 36 |
pipe = AutoPipelineForInpainting.from_pretrained(
|
| 37 |
"neonforestmist/Clover-Image-Tiny-Inpaint",
|
| 38 |
torch_dtype="auto",
|
| 39 |
)
|
|
|
|
| 40 |
image = pipe(
|
| 41 |
prompt="a tiny glass greenhouse glowing in a moonlit garden",
|
| 42 |
image=load_image("input.png"),
|
| 43 |
mask_image=load_image("mask.png"),
|
| 44 |
-
num_inference_steps=
|
|
|
|
| 45 |
).images[0]
|
| 46 |
image.save("clover-inpaint.png")
|
| 47 |
```
|
|
@@ -53,6 +55,21 @@ The mask is a grayscale image: white means “regenerate” and black means
|
|
| 53 |
- `remove the person from the masked area and continue the background naturally`
|
| 54 |
- `add a red enamel kettle on the masked countertop`
|
| 55 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
For repeatable experiments, keep the input image, mask, seed, scheduler, and
|
| 57 |
step count together. Inpainting is local editing: the unmasked region is
|
| 58 |
provided as the masked-image conditioning and is also preserved by the native
|
|
@@ -73,6 +90,14 @@ Conversion and the native iOS integration live in the source Clover repo:
|
|
| 73 |
- [`Clover-iOS`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/tree/main/Clover-iOS)
|
| 74 |
- [`training/README-INPAINTING.md`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/training/README-INPAINTING.md)
|
| 75 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
## Training provenance
|
| 77 |
|
| 78 |
Training uses synthetic rectangle, ellipse, and brush masks over the pinned
|
|
|
|
| 30 |
## Diffusers example
|
| 31 |
|
| 32 |
```python
|
| 33 |
+
from diffusers import AutoPipelineForInpainting, DPMSolverMultistepScheduler
|
| 34 |
from diffusers.utils import load_image
|
| 35 |
|
| 36 |
pipe = AutoPipelineForInpainting.from_pretrained(
|
| 37 |
"neonforestmist/Clover-Image-Tiny-Inpaint",
|
| 38 |
torch_dtype="auto",
|
| 39 |
)
|
| 40 |
+
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
|
| 41 |
image = pipe(
|
| 42 |
prompt="a tiny glass greenhouse glowing in a moonlit garden",
|
| 43 |
image=load_image("input.png"),
|
| 44 |
mask_image=load_image("mask.png"),
|
| 45 |
+
num_inference_steps=20,
|
| 46 |
+
padding_mask_crop=64,
|
| 47 |
).images[0]
|
| 48 |
image.save("clover-inpaint.png")
|
| 49 |
```
|
|
|
|
| 55 |
- `remove the person from the masked area and continue the background naturally`
|
| 56 |
- `add a red enamel kettle on the masked countertop`
|
| 57 |
|
| 58 |
+
## Inpainting example
|
| 59 |
+
|
| 60 |
+
The example below uses DPM-Solver++, 20 steps, CFG 7.5, and seed 1337. The
|
| 61 |
+
prompt is `a realistic orange cat sitting in the doorway, detailed photography`.
|
| 62 |
+
Only the white doorway mask is replaced.
|
| 63 |
+
|
| 64 |
+
| Source | White mask | Result |
|
| 65 |
+
|:---:|:---:|:---:|
|
| 66 |
+
|  |  |  |
|
| 67 |
+
|
| 68 |
+
For small masks, `padding_mask_crop=64` gives the edit more useful latent
|
| 69 |
+
resolution before Diffusers maps it back into the source image. DPM-Solver++
|
| 70 |
+
is the recommended scheduler for this release; PNDM can become unstable at
|
| 71 |
+
higher step counts. Start at 20 steps and keep interactive runs at or below 50.
|
| 72 |
+
|
| 73 |
For repeatable experiments, keep the input image, mask, seed, scheduler, and
|
| 74 |
step count together. Inpainting is local editing: the unmasked region is
|
| 75 |
provided as the masked-image conditioning and is also preserved by the native
|
|
|
|
| 90 |
- [`Clover-iOS`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/tree/main/Clover-iOS)
|
| 91 |
- [`training/README-INPAINTING.md`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/training/README-INPAINTING.md)
|
| 92 |
|
| 93 |
+
## LoRA compatibility
|
| 94 |
+
|
| 95 |
+
Diffusers exposes LoRA loading for inpainting pipelines, but an adapter must be
|
| 96 |
+
trained against this model's 9-channel U-Net. Existing Clover Image Tiny style
|
| 97 |
+
LoRAs target the Regular 4-channel U-Net and are not interchangeable. The
|
| 98 |
+
current Core ML export does not expose dynamic LoRA state; an inpainting-specific
|
| 99 |
+
adapter can instead be fused and converted as a separate bundle.
|
| 100 |
+
|
| 101 |
## Training provenance
|
| 102 |
|
| 103 |
Training uses synthetic rectangle, ellipse, and brush masks over the pinned
|
examples/mask-doorway.png
ADDED
|
examples/result-cat.png
ADDED
|
Git LFS Details
|
examples/source-greenhouse.png
ADDED
|
Git LFS Details
|