Image-to-Image
Diffusers
Safetensors
Core ML
StableDiffusionInpaintPipeline
clover-image
inpainting
stable-diffusion
Instructions to use neonforestmist/Clover-Image-Tiny-Inpaint with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use neonforestmist/Clover-Image-Tiny-Inpaint with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline from diffusers.utils import load_image # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("neonforestmist/Clover-Image-Tiny-Inpaint", dtype=torch.bfloat16, device_map="cuda") prompt = "Turn this cat into a dog" input_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png") image = pipe(image=input_image, prompt=prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Add Modal training and inpainting model scaffold
Browse files
training/README-INPAINTING.md
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Clover Image Tiny Inpainting
|
| 2 |
+
|
| 3 |
+
The inpainting model keeps Clover’s text encoder, VAE, scheduler, and safety
|
| 4 |
+
checker, and replaces only the U-Net with a 9-channel fine-tuned version.
|
| 5 |
+
|
| 6 |
+
The U-Net input is ordered as:
|
| 7 |
+
|
| 8 |
+
```text
|
| 9 |
+
[noisy_latent (4), mask (1), masked_image_latent (4)]
|
| 10 |
+
```
|
| 11 |
+
|
| 12 |
+
White mask pixels are regenerated; black pixels are preserved. Masks are
|
| 13 |
+
sampled procedurally during training, so the dataset only needs image/caption
|
| 14 |
+
pairs. The pinned default dataset is `prithivMLmods/Caption3o-Opt` at revision
|
| 15 |
+
`17e893f785fcd3f5d6fc4a5d65a914b9f7b1ff5b` (Apache-2.0, `image` + `caption`).
|
| 16 |
+
|
| 17 |
+
## Modal
|
| 18 |
+
|
| 19 |
+
The local Modal CLI is already configured for the `guccichungus69` profile.
|
| 20 |
+
Use that profile before running the commands below:
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
modal profile activate guccichungus69
|
| 24 |
+
modal run modal_inpaint.py --smoke --steps 2
|
| 25 |
+
modal run modal_inpaint.py --steps 4000
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
The job uses one A10G and writes the finished Diffusers pipeline to the
|
| 29 |
+
`clover-image-tiny-inpaint-output` Volume. No Hub token is required for the
|
| 30 |
+
default public dataset. The estimated compute ceiling for the default four-hour
|
| 31 |
+
job is about $4 at $1/hour; the actual charge depends on runtime.
|
| 32 |
+
|
| 33 |
+
Download a completed output directory with:
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
modal volume get clover-image-tiny-inpaint-output \
|
| 37 |
+
clover-image-tiny-inpaint \
|
| 38 |
+
./artifacts/clover-image-tiny-inpaint
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
If a job stops before writing its final pipeline, the Volume retains any files
|
| 42 |
+
that were committed. The current trainer saves only at the end, so use a
|
| 43 |
+
separate `--output-name` for retries.
|
| 44 |
+
|
| 45 |
+
## Local dry run
|
| 46 |
+
|
| 47 |
+
The same trainer can run locally when the model and dataset are available:
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
python inpainting/train.py \
|
| 51 |
+
--pretrained_model_name_or_path . \
|
| 52 |
+
--dataset_name prithivMLmods/Caption3o-Opt \
|
| 53 |
+
--dataset_revision 17e893f785fcd3f5d6fc4a5d65a914b9f7b1ff5b \
|
| 54 |
+
--max_train_samples 4 \
|
| 55 |
+
--max_train_steps 2 \
|
| 56 |
+
--output_dir /tmp/clover-image-tiny-inpaint-smoke
|
| 57 |
+
```
|
| 58 |
+
|
| 59 |
+
The complete model is large enough that Modal is the intended training
|
| 60 |
+
environment. The current repo’s Apple Core ML environment is for conversion,
|
| 61 |
+
not PyTorch training.
|