mokshhere/skin-lesion-boundary-diffusion

Skin-Lesion Boundary Diffusion — Escalation Study Checkpoints

Four diffusion checkpoints from a measurement study on whether mask-conditioned diffusion synthesis can reproduce the gradual boundaries real dermoscopic lesions have. Short answer: not well enough — see the escalation results below and the full write-up.

• Code: https://github.com/ishaannk/skin-lesion-seg
• Paper draft: paper/paper_draft.md in the code repo
• Shareable summary: paper/share/boundary_gap_report.html in the code repo (open in any browser)

All checkpoints are fine-tuned only on the ISIC 2018 Task 1 official training split (2594 images) — never on validation or test images.

What's in this repo

Folder	What it is	Base	Fine-tune objective
sd_lesion_full/unet	Lesion-painting UNet, Level 1 (hard-mask pipeline)	SD 1.5	region-masked reconstruction inside a binary lesion mask
sd_background_full/unet	Healthy-skin restoration UNet (removes real lesions to make lesion-free backdrops)	SD 1.5	region-masked reconstruction outside the mask
sd_lesion_alphacond/unet	Lesion-painting UNet, Level 3 (generation-time soft-boundary conditioning)	SD 1.5	same objective, plus a widened conv_in (zero-init extra channel) fed a per-image soft alpha ramp at every step, so the network is asked to paint a gradual boundary rather than have one composited on afterward
mask_ddpm_full/unet	64×64 DDPM over lesion masks, used to sample guide-mask shapes for synthesis	trained from scratch (not SD-derived)	standard DDPM denoising objective over binary masks

Headline result (why "boundary" is in every filename)

Real lesion boundaries fade over a mean 18.65 px transition width (measured directly from ISIC photographs). Three escalating fixes were tried; none closed the gap:

Level	Mean width	vs. real	Closes the gap?
Real (ground truth)	18.65 px	—	—
L1 — hard mask + fixed feather (sd_lesion_full)	16.25 px	1.15× narrower	no
L2 — post-hoc soft-alpha compositing (sd_lesion_full, different sampling code)	16.82 px	1.11× narrower	no
L3 — generation-time conditioning (sd_lesion_alphacond)	16.99 px	1.10× narrower	31% of the L1 gap closed, 69% remains

Downstream: real-only segmentation training beat every synthetic-augmented configuration tried with these generators (three architectures in a row across the full project). See the code repo for the full escalation study, the VAE round-trip control that rules out the autoencoder as the cause, and the memorisation audit (max similarity to real test images 0.946–0.961, ≤1 near-duplicate in 2000 generated pairs at every level).

Usage

from diffusers import UNet2DConditionModel

# Level 1 lesion UNet (standard 4-channel latent input)
unet = UNet2DConditionModel.from_pretrained(
    "mokshhere/skin-lesion-boundary-diffusion", subfolder="sd_lesion_full/unet"
)

# Level 3 alpha-conditioned UNet — NOTE: conv_in takes 5 channels (latent + soft alpha
# ramp concatenated), not the standard 4. See src/diffusion/generate.py in the code repo
# (region_blended_sample) for the exact conditioning procedure at inference time.
unet_l3 = UNet2DConditionModel.from_pretrained(
    "mokshhere/skin-lesion-boundary-diffusion", subfolder="sd_lesion_alphacond/unet"
)
print(unet_l3.config.in_channels)  # 5

The mask DDPM uses a different loader (src/diffusion/mask_ddpm.py:sample_masks) — see the code repo, it is not a standard diffusers pipeline component.

License

Fine-tuned from stable-diffusion-v1-5/stable-diffusion-v1-5, released under the CreativeML Open RAIL-M license. These checkpoints inherit the same license and its use-based restrictions (see LICENSE in the base model's repo). mask_ddpm_full is trained from scratch on binary lesion masks only (no natural-image weights) and carries no such inheritance, but is released under the same license for consistency with the rest of this collection.

Not intended for clinical use

These are research artifacts from a negative-result measurement study, evaluated only on ISIC 2018 (a public, de-identified dermoscopy benchmark). Do not use for diagnosis or any clinical decision-making.