Add model card

README.md

@@ -0,0 +1,173 @@
+---
+license: mit
+tags:
+  - stable-diffusion-xl
+  - sdxl
+  - ip-adapter
+  - product-photography
+  - e-commerce
+  - text-to-image
+base_model: stabilityai/stable-diffusion-xl-base-1.0
+library_name: diffusers
+---
+
+# StudioDiffusion IP-Adapter (Shopify / Etsy / eBay)
+
+Three **IP-Adapter** weight sets trained on top of [Stable Diffusion XL](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0), each targeting a distinct e-commerce platform aesthetic:
+
+- **Shopify** — clean white / neutral backgrounds, studio lighting, minimal props, high contrast subject separation.
+- **Etsy** — warm color temperature, lifestyle / craft props, natural light, textured surfaces, artisanal hand-crafted feel.
+- **eBay** — bright even lighting, plain or gradient background, sharp focus on subject, utilitarian clarity.
+
+Companion code and training pipeline: **https://github.com/s-zx/StudioDiffusion**
+
+## Repository layout
+
+| Path | Contents |
+|---|---|
+| `shopify/final/{image_proj_model,ip_attn_processors}.pt` | Shopify checkpoint @ step 3000 |
+| `shopify/train.log` | Shopify val-loss per 250 steps |
+| `etsy/final/{image_proj_model,ip_attn_processors}.pt` | Etsy checkpoint @ step 3000 |
+| `etsy/checkpoint-500/{image_proj_model,ip_attn_processors}.pt` | **Recommended** Etsy checkpoint — best val loss, before mild overfit |
+| `etsy/train.log` | Etsy val-loss per 250 steps |
+| `ebay/final/{image_proj_model,ip_attn_processors}.pt` | eBay checkpoint @ step 3000 |
+| `ebay/train.log` | eBay val-loss per 250 steps |
+
+Each checkpoint follows the `IPAdapterSDXL.save_pretrained` format defined in [`adapters/ip_adapter/model.py`](https://github.com/s-zx/StudioDiffusion/blob/main/adapters/ip_adapter/model.py). Two files per checkpoint: `image_proj_model.pt` (CLIP-embed → token projection) and `ip_attn_processors.pt` (injected K/V weights for every cross-attention block of the SDXL UNet).
+
+## Usage
+
+### Download
+
+```python
+from huggingface_hub import snapshot_download
+
+# Full set (~5.6 GB)
+snapshot_download(
+    repo_id="jasonshen8848/StudioDiffusion-ip-adapter",
+    local_dir="checkpoints/ip_adapter",
+)
+
+# Single platform (~1.4 GB)
+snapshot_download(
+    repo_id="jasonshen8848/StudioDiffusion-ip-adapter",
+    local_dir="checkpoints/ip_adapter",
+    allow_patterns=["shopify/final/*", "shopify/train.log"],
+)
+```
+
+### Generate — minimal inference example
+
+A complete working example is at [`inference/smoke.py`](https://github.com/s-zx/StudioDiffusion/blob/main/inference/smoke.py). Core pattern:
+
+```python
+import torch
+from diffusers import StableDiffusionXLPipeline, AutoencoderKL
+from PIL import Image
+from torchvision import transforms
+
+from adapters.ip_adapter.model import IPAdapterSDXL  # from the GitHub repo
+
+device, dtype = "mps", torch.float16  # also works on CUDA with these
+
+pipe = StableDiffusionXLPipeline.from_pretrained(
+    "stabilityai/stable-diffusion-xl-base-1.0",
+    vae=AutoencoderKL.from_pretrained(
+        "madebyollin/sdxl-vae-fp16-fix", torch_dtype=dtype,
+    ),
+    torch_dtype=dtype,
+).to(device)
+
+adapter = IPAdapterSDXL.load_pretrained(
+    unet=pipe.unet,
+    load_directory="checkpoints/ip_adapter/shopify/final",
+    image_encoder_id="openai/clip-vit-large-patch14-336",
+    num_tokens=16,
+    adapter_scale=1.0,
+).to(device=device, dtype=dtype)
+
+clip_transform = transforms.Compose([
+    transforms.Resize(336, interpolation=transforms.InterpolationMode.BICUBIC),
+    transforms.CenterCrop(336),
+    transforms.ToTensor(),
+    transforms.Normalize(
+        mean=[0.48145466, 0.4578275, 0.40821073],
+        std=[0.26862954, 0.26130258, 0.27577711],
+    ),
+])
+
+ref = Image.open("my_product.jpg").convert("RGB")
+clip_input = clip_transform(ref).unsqueeze(0).to(device=device, dtype=dtype)
+with torch.no_grad():
+    cond_ip, uncond_ip = adapter.encode_image(clip_input)
+ip_hidden_states = torch.cat([uncond_ip, cond_ip], dim=0)  # [uncond, cond] for CFG
+
+image = pipe(
+    prompt="a professional product photograph",
+    negative_prompt="blurry, low quality, distorted, artifacts",
+    num_inference_steps=30,
+    guidance_scale=7.5,
+    height=512, width=512,
+    cross_attention_kwargs={"ip_hidden_states": ip_hidden_states},
+).images[0]
+image.save("out.png")
+```
+
+## Training summary
+
+| | Shopify | Etsy | eBay |
+|---|---|---|---|
+| Train images | 353 | 325 | 518 |
+| Val images | 88 | 81 | 129 |
+| Start val loss (step 250) | 0.073747 | 0.131454 | 0.058868 |
+| End val loss (step 3000) | 0.072500 | 0.132335 | 0.055920 |
+| Best val loss | 0.072463 @ step 2000 | **0.131412 @ step 750** | 0.055920 @ step 3000 |
+| Δ val loss | **−1.7%** ↓ | **+0.7%** ↑ (mild overfit) | **−5.0%** ↓ |
+| Wall-clock | ~9 h | ~9 h | ~9 h |
+
+**Hyperparameters** (identical across platforms):
+
+- Base: `stabilityai/stable-diffusion-xl-base-1.0`
+- VAE: `madebyollin/sdxl-vae-fp16-fix`
+- Image encoder: `openai/clip-vit-large-patch14-336` (frozen)
+- Optimizer: AdamW, lr=1e-4, (β₁, β₂)=(0.9, 0.999), wd=0.01
+- LR schedule: cosine with 200-step warmup
+- **Mixed precision: "no" (pure fp32)** — required for MPS stability
+- Image size: 512×512 diffusion path; 336×336 CLIP-branch (fixed by encoder)
+- Effective batch: 2 micro × 4 grad-accum = 8
+- Steps: 3000 (= ~75 epochs on Shopify/Etsy, ~46 on eBay)
+- Gradient checkpointing: enabled (required on 48 GB M4 Pro)
+- Seed: 42
+
+**Training data**: curated via `data/curate_platform.py` in the companion repo. Sources: Amazon Berkeley Objects (ABO), LAION-Aesthetics, DeepFashion2. ~400 images per platform selected by CLIP platform-prompt similarity + category balancing; 80/20 train/val split recorded in manifest CSVs.
+
+**Hardware**: Apple MacBook Pro M4 Pro, 48 GB unified memory, PyTorch MPS backend.
+
+## Known limitations
+
+- **Captions are identity placeholders.** Training used `"a product photo"` for every sample (BLIP-2 caption generation was deferred). Text conditioning therefore provides minimal per-sample variance; all platform aesthetic signal flows through the IP-Adapter image branch.
+- **Shopify adapter may over-desaturate color.** In qualitative spot checks, the Shopify adapter can push outputs towards white even when the reference product has a distinct color. If color fidelity matters, try `adapter_scale=0.5–0.75` at inference.
+- **Etsy is mildly overfit after step 750.** Val loss rose ~0.7% from step 750 → 3000. The `final/` checkpoint is stylistically the strongest but diverges more from the reference content. **For content-preserving generation, prefer `etsy/checkpoint-500/`** (closest available to the val-loss optimum).
+- **fp32 training was forced by MPS.** On Apple Silicon, autocast fp16/bf16 for SDXL + IP-Adapter raises an MPS `NDArrayMatrixMultiplication` assertion on the first forward pass. These weights are architecturally compatible with fp16 inference (verified on MPS — see the example above), but **fp16 / bf16 training** of this adapter configuration on CUDA has not been tested here.
+- **No ControlNet / segmentation integration in these weights.** The companion repo plans a SAM2 + seg-trained ControlNet path; these checkpoints were trained without any spatial conditioning signal.
+
+## License
+
+MIT — matches the parent project.
+
+Individual dataset licenses (ABO CC BY-NC 4.0, DeepFashion2 gated, LAION CC BY 4.0) apply to the *training data*, not to these weight files. Please consult those upstream licenses before commercial use.
+
+## Citation
+
+If you use these checkpoints, please cite the parent project:
+
+```bibtex
+@misc{studiodiffusion2026,
+  title  = {StudioDiffusion: Training Platform-Specific Aesthetic Adapters for Product
+            Photography Using Segmentation-Conditioned Diffusion Models},
+  author = {Shen, Jason and contributors},
+  year   = {2026},
+  howpublished = {\url{https://github.com/s-zx/StudioDiffusion}},
+  note   = {CS 7643 Deep Learning final project, Georgia Tech}
+}
+```