training/README-INPAINTING.md

# Clover Image Tiny Inpainting

The inpainting model keeps Clover’s text encoder, VAE, scheduler, and safety
checker, and replaces only the U-Net with a 9-channel fine-tuned version.

The U-Net input is ordered as:

```text
[noisy_latent (4), mask (1), masked_image_latent (4)]
```

White mask pixels are regenerated; black pixels are preserved. The v2 recipe
warm-starts from the released nine-channel checkpoint and distills the pinned
official `stable-diffusion-v1-5/stable-diffusion-inpainting` U-Net. A smaller
ground-truth diffusion term prevents the student from merely copying teacher
errors. Min-SNR timestep weighting and a spatial objective emphasize the
masked region, its boundary, and enough unmasked context to learn coherent
transitions.

Masks are sampled procedurally as free-form strokes, multiple disconnected
strokes, rounded boxes, ellipses, polygons, multiple object-like regions, and
outpainting bands. The pinned dataset is `prithivMLmods/Caption3o-Opt` at
revision `17e893f785fcd3f5d6fc4a5d65a914b9f7b1ff5b` (Apache-2.0, `image` +
`caption`). A deterministic 128-image tail is excluded from training and used
for same-mask, same-seed checkpoint comparison.

## Modal

The local Modal CLI is already configured for the `guccichungus69` profile.
Use that profile before running the commands below:

```bash
modal profile activate guccichungus69
modal run modal_inpaint.py --smoke --steps 1
modal run --detach modal_inpaint.py \
  --steps 500 \
  --lr-warmup-steps 25 \
  --output-name clover-image-tiny-inpaint-v2-pilot-500
```

The job uses one A10 and writes the finished Diffusers pipeline to the
`clover-image-tiny-inpaint-output` Volume. No Hub token is required for the
default public dataset. Resumable Accelerator checkpoints are committed to the
persistent Volume. A separate cache avoids downloading the student, teacher,
and dataset again on every run.

The published v2 checkpoint is the 500-step cosine-decay sweep. A longer run
was rejected after its held-out reconstruction and semantic gates regressed
into colored high-frequency artifacts. The release pins the best validated
checkpoint rather than the final checkpoint from the largest job.

Download a completed output directory with:

```bash
modal volume get clover-image-tiny-inpaint-output \
  clover-image-tiny-inpaint-v2-pilot-500/unet \
  ./artifacts/clover-image-tiny-inpaint-v2
```

If a job stops after a committed checkpoint, resume the same output with
`--resume`. Completed output directories are immutable to the launcher; use a
new `--output-name` for a distinct experiment.

## Local dry run

The same trainer can run locally when the model and dataset are available:

```bash
python inpainting/train.py \
  --pretrained_model_name_or_path . \
  --dataset_name prithivMLmods/Caption3o-Opt \
  --dataset_revision 17e893f785fcd3f5d6fc4a5d65a914b9f7b1ff5b \
  --max_train_samples 4 \
  --max_train_steps 2 \
  --output_dir /tmp/clover-image-tiny-inpaint-smoke
```

The complete model is large enough that Modal is the intended training
environment. The current repo’s Apple Core ML environment is for conversion,
not PyTorch training.