neonforestmist commited on
Commit
8858ec2
·
verified ·
1 Parent(s): 711a588

Add validated inpainting example and stable sampler guidance

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ examples/result-cat.png filter=lfs diff=lfs merge=lfs -text
37
+ examples/source-greenhouse.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -30,18 +30,20 @@ the VAE encoder needed to prepare the masked-image latent on device.
30
  ## Diffusers example
31
 
32
  ```python
33
- from diffusers import AutoPipelineForInpainting
34
  from diffusers.utils import load_image
35
 
36
  pipe = AutoPipelineForInpainting.from_pretrained(
37
  "neonforestmist/Clover-Image-Tiny-Inpaint",
38
  torch_dtype="auto",
39
  )
 
40
  image = pipe(
41
  prompt="a tiny glass greenhouse glowing in a moonlit garden",
42
  image=load_image("input.png"),
43
  mask_image=load_image("mask.png"),
44
- num_inference_steps=30,
 
45
  ).images[0]
46
  image.save("clover-inpaint.png")
47
  ```
@@ -53,6 +55,21 @@ The mask is a grayscale image: white means “regenerate” and black means
53
  - `remove the person from the masked area and continue the background naturally`
54
  - `add a red enamel kettle on the masked countertop`
55
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
  For repeatable experiments, keep the input image, mask, seed, scheduler, and
57
  step count together. Inpainting is local editing: the unmasked region is
58
  provided as the masked-image conditioning and is also preserved by the native
@@ -73,6 +90,14 @@ Conversion and the native iOS integration live in the source Clover repo:
73
  - [`Clover-iOS`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/tree/main/Clover-iOS)
74
  - [`training/README-INPAINTING.md`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/training/README-INPAINTING.md)
75
 
 
 
 
 
 
 
 
 
76
  ## Training provenance
77
 
78
  Training uses synthetic rectangle, ellipse, and brush masks over the pinned
 
30
  ## Diffusers example
31
 
32
  ```python
33
+ from diffusers import AutoPipelineForInpainting, DPMSolverMultistepScheduler
34
  from diffusers.utils import load_image
35
 
36
  pipe = AutoPipelineForInpainting.from_pretrained(
37
  "neonforestmist/Clover-Image-Tiny-Inpaint",
38
  torch_dtype="auto",
39
  )
40
+ pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
41
  image = pipe(
42
  prompt="a tiny glass greenhouse glowing in a moonlit garden",
43
  image=load_image("input.png"),
44
  mask_image=load_image("mask.png"),
45
+ num_inference_steps=20,
46
+ padding_mask_crop=64,
47
  ).images[0]
48
  image.save("clover-inpaint.png")
49
  ```
 
55
  - `remove the person from the masked area and continue the background naturally`
56
  - `add a red enamel kettle on the masked countertop`
57
 
58
+ ## Inpainting example
59
+
60
+ The example below uses DPM-Solver++, 20 steps, CFG 7.5, and seed 1337. The
61
+ prompt is `a realistic orange cat sitting in the doorway, detailed photography`.
62
+ Only the white doorway mask is replaced.
63
+
64
+ | Source | White mask | Result |
65
+ |:---:|:---:|:---:|
66
+ | ![Greenhouse source](examples/source-greenhouse.png) | ![White doorway mask](examples/mask-doorway.png) | ![Cat inpainted into the doorway](examples/result-cat.png) |
67
+
68
+ For small masks, `padding_mask_crop=64` gives the edit more useful latent
69
+ resolution before Diffusers maps it back into the source image. DPM-Solver++
70
+ is the recommended scheduler for this release; PNDM can become unstable at
71
+ higher step counts. Start at 20 steps and keep interactive runs at or below 50.
72
+
73
  For repeatable experiments, keep the input image, mask, seed, scheduler, and
74
  step count together. Inpainting is local editing: the unmasked region is
75
  provided as the masked-image conditioning and is also preserved by the native
 
90
  - [`Clover-iOS`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/tree/main/Clover-iOS)
91
  - [`training/README-INPAINTING.md`](https://huggingface.co/neonforestmist/Clover-Image-Tiny/blob/main/training/README-INPAINTING.md)
92
 
93
+ ## LoRA compatibility
94
+
95
+ Diffusers exposes LoRA loading for inpainting pipelines, but an adapter must be
96
+ trained against this model's 9-channel U-Net. Existing Clover Image Tiny style
97
+ LoRAs target the Regular 4-channel U-Net and are not interchangeable. The
98
+ current Core ML export does not expose dynamic LoRA state; an inpainting-specific
99
+ adapter can instead be fused and converted as a separate bundle.
100
+
101
  ## Training provenance
102
 
103
  Training uses synthetic rectangle, ellipse, and brush masks over the pinned
examples/mask-doorway.png ADDED
examples/result-cat.png ADDED

Git LFS Details

  • SHA256: 63469782e1dae9e655d478a47d184e59ef56ed144979b93fc67c2fd5f32b1b2a
  • Pointer size: 131 Bytes
  • Size of remote file: 835 kB
examples/source-greenhouse.png ADDED

Git LFS Details

  • SHA256: 4cfaa244af894d85af35fb58e57b2c626513574da6f8a77016f159995d99d1a2
  • Pointer size: 131 Bytes
  • Size of remote file: 607 kB