File size: 30,874 Bytes
30cbfdc 3d958ad 30cbfdc 3d958ad 30cbfdc 3d958ad 30cbfdc 2c1a715 30cbfdc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 | ---
language:
- en
library_name: "diffusers"
pipeline_tag: "image-to-image"
tags:
- "collage-diffusion"
- "image-generation"
- "wacv-2024"
datasets: []
base_model: "sd2-community/stable-diffusion-2-1-base"
---
<!-- --8<-- [start:card] -->
# Model Card for Collage Diffusion
[](https://arxiv.org/abs/2303.00262)
This package exposes Collage Diffusion as a [`🧨diffusers`](https://huggingface.co/docs/diffusers/index) pipeline that harmonizes text-described RGBA layers into a single image.
## Model Details
### Model Description
Collage Diffusion combines a scene prompt with ordered RGBA image layers. Visible alpha masks guide cross-attention to each layer's text, while per-layer noise levels control how much of each source layer is preserved during diffusion. The pipeline can load caller-provided textual-inversion embeddings. It uses the upstream Stable Diffusion 2.1 base checkpoint and does not train, store, or publish model-specific weights.
- **Developed by:** Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian.
- **Shared by:** creative-graphic-design.
- **Model type:** image generation; task: image-to-image; conditioning: layers, text.
- **Language(s) (NLP):** English scene and layer prompts.
- **License:** Apache-2.0 for this package; the base checkpoint and input assets retain their own terms.
- **Finetuned from:** No Collage Diffusion checkpoint; inference loads `sd2-community/stable-diffusion-2-1-base` at the pinned revision below under the CreativeML Open RAIL++-M terms declared by its model card.
### Model Sources
- **Repository:** [Collage Diffusion source repository](https://github.com/VSAnimator/collage-diffusion); it has no declared repository license. Its notebook is used as a behavioral reference; its code is not included here. Notebook-linked image layers and textual-inversion embeddings are archived, with source links and integrity records, in the [model repository's source-assets directory](https://huggingface.co/creative-graphic-design/collage-diffusion/tree/main/source-assets).
- **Paper:** [Collage Diffusion](https://arxiv.org/abs/2303.00262).
- **Related implementation:** [Collage Diffusion UI](https://github.com/linden-li/collage-diffusion-ui) was inspected as a secondary reference. Its repository also has no declared license; its code is not included here.
- **Released weights:** The package does not contain its own weights. It loads the [Stable Diffusion 2.1 base mirror](https://huggingface.co/sd2-community/stable-diffusion-2-1-base) at a pinned revision.
### Base Checkpoint Provenance and Integrity
The authors' notebook names `stabilityai/stable-diffusion-2-1-base`, but that Hub repository now returns HTTP 404. The comparison and package examples use `sd2-community/stable-diffusion-2-1-base` at revision `4e63672c03103b6c636b8fb4119ba982469b2955`. The model card declares the CreativeML Open RAIL++-M license; users must review and follow those terms.
The main weight files were checked across three available copies. Their SHA-256 values match byte-for-byte:
| Copy | Revision | UNet SHA-256 | VAE SHA-256 | Text encoder SHA-256 |
| -------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------ | ------------------------------------------------------------------ | ------------------------------------------------------------------ |
| `sd2-community/stable-diffusion-2-1-base` | `4e63672c03103b6c636b8fb4119ba982469b2955` | `6dfae3e5f7d459b50f4b0850ead945972c75bb0e1897628933e169eb43974214` | `a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815` | `cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c` |
| `Manojb/stable-diffusion-2-1-base` | `0094d483a120f3f33dafbd187ea4aa60d10de75c` | same | same | same |
| `LanguageMachines/stable-diffusion-2-1-base` | `2579f99c90b5a3019fc115ee9566d5ec4f17fccf` | same | same | same |
SHA-256 values for every file used from the selected pinned snapshot:
| File | SHA-256 |
| -------------------------------------------- | ------------------------------------------------------------------ |
| `model_index.json` | `518553e098fdaa4153d9a92139e344e44f53bb329c22f383293e3099c4f1bad2` |
| `feature_extractor/preprocessor_config.json` | `2a1da83b5e1032aaeef397552ddb408dca0d8cd1dc58f61bf6abf38d6f33a0a2` |
| `scheduler/scheduler_config.json` | `11ac5627d7df0fa344b875c4b5722b1767a8a2aa1684c2cf8b4d614300127234` |
| `text_encoder/config.json` | `7e33bfc475f6cf76e69b28b101993ca834d1f22ff4d35caefde32d63d4f42fba` |
| `text_encoder/model.safetensors` | `cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c` |
| `tokenizer/merges.txt` | `9fd691f7c8039210e0fced15865466c65820d09b63988b0174bfe25de299051a` |
| `tokenizer/special_tokens_map.json` | `f118ab3a983206e4f32583448de6bd6aae4ee21869135cef1f5848a753cdaab6` |
| `tokenizer/tokenizer_config.json` | `d562b18a0c6d32a168fce3d7a342fce64579ebacffb98fb71b0affc494fc32d8` |
| `tokenizer/vocab.json` | `e089ad92ba36837a0d31433e555c8f45fe601ab5c221d4f607ded32d9f7a4349` |
| `unet/config.json` | `ce0c6d379e3b1d3e1f79338de70c80a1e36f17a6372439a8249b6fb0dfa1b608` |
| `unet/diffusion_pytorch_model.safetensors` | `6dfae3e5f7d459b50f4b0850ead945972c75bb0e1897628933e169eb43974214` |
| `vae/config.json` | `424117cb534ce03497c41305ed868980123917b2b6abba4bbaa615e968772903` |
| `vae/diffusion_pytorch_model.safetensors` | `a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815` |
Both sides of the notebook comparison use this same pinned snapshot and the file hashes listed above. The selected model copy and its weights are not redistributed by this package.
## Supported Checkpoints
| Artifact | Hub ID | Status |
| ------------------------- | ------------------------------------------- | ----------------------------------------- |
| `diffusers` pipeline code | `creative-graphic-design/collage-diffusion` | planned, code only; no learned checkpoint |
When publishing this pipeline to the Hub, its `model_index.json` and `scheduler/scheduler_config.json` must specify `EulerAncestralDiscreteScheduler`.
## Uses
### Direct Use
Use the pipeline to harmonize a layered collage while guiding object placement through masks and controlling source preservation through per-layer noise levels.
```python
from PIL import Image
from diffusers import EulerAncestralDiscreteScheduler
from collage_diffusion import CollageDiffusionPipeline, CollageLayer
base = "sd2-community/stable-diffusion-2-1-base"
revision = "4e63672c03103b6c636b8fb4119ba982469b2955"
# Cross-attention guidance uses the scheduler's sigma values.
scheduler = EulerAncestralDiscreteScheduler.from_pretrained(
base, subfolder="scheduler", revision=revision
)
pipe = CollageDiffusionPipeline.from_pretrained(
base, revision=revision, scheduler=scheduler
)
layers = [
CollageLayer(
image=Image.open("toy.png").convert("RGBA"),
prompt="a small wooden toy",
position=(64, 96),
noise_level=0.35,
positive_attention_weight=0.5,
negative_attention_weight=0.5,
),
]
result = pipe(
prompt="a small wooden toy on a studio table",
layers=layers,
negative_prompt="collage",
seed=0,
num_inference_steps=20,
)
result.images[0].save("harmonized.png")
print("Saved harmonized.png")
```
```text
Saved harmonized.png
```
Textual-inversion embeddings are caller-supplied with a token-to-file mapping, and the matching token is set on its `CollageLayer`. ControlNet and textual-inversion training are outside this package's current scope.
### Examples
#### Shared setup
These examples follow the [Collage Diffusion notebook](https://github.com/VSAnimator/collage-diffusion/blob/main/CollageDiffusionDemo.ipynb) and its [Google Drive image layers](https://drive.google.com/drive/folders/1EfoLfWbn5fl7c092uk96zHtJOVHRlIqI?usp=sharing) and [textual-inversion assets](https://drive.google.com/drive/folders/13Gg8y-0FxdPcuhrPOUw6Ze9wa0Ruu-cI?usp=sharing). Because the Drive files may disappear, copies and their source URLs, retrieval dates, sizes, and SHA-256 values are recorded in the [model repository's source-assets directory](https://huggingface.co/creative-graphic-design/collage-diffusion/tree/main/source-assets). The original author, rights holder, and per-file license are unknown; the documentation-images dataset's Apache-2.0 license does not apply to those files.
Each image places the input collage on the left and the generated result on the right. All scenes use `sd2-community/stable-diffusion-2-1-base` at revision `4e63672c03103b6c636b8fb4119ba982469b2955`, `EulerAncestralDiscreteScheduler`, seed `0`, 512 × 512 resolution, guidance scale `7.5`, and 50 requested inference steps. The active step count is 40 at strength `0.8` and 37 at `0.75`. The base checkpoint uses the [CreativeML Open RAIL++-M License](https://huggingface.co/sd2-community/stable-diffusion-2-1-base). The code uses every CPU thread available to its runtime; the generated examples used **2 threads**.
#### Toy Box
Prompt: `a teddy bear, a wood train, and an american football, in front of a tan background`. Negative prompt: `collage`. Strength: `0.8`; `noise_blur=20`; 50 requested (40 active) steps.
#### Bento Box
Prompt: `a bento box with rice, edamame, ginger, and sushi`. Negative prompt: `collage`. Strength: `0.8`; `noise_blur=20`; 50 requested (40 active) steps.
#### Cake Table
Prompt: `a wood table with two white chairs behind, two pink cakes on top, maroon bookshelves behind, and winter window`. Negative prompt: `collage, warped, cupcake`. Strength: `0.75`; `noise_blur=10`; 50 requested (37 active) steps.
The following script builds each scene from its archived image layers and textual-inversion embeddings, then saves the side-by-side examples above. It generates all three by default; set `COLLAGE_SCENES` to a comma-separated subset of scene names to run only selected examples.
```python
import gc
import os
import time
import torch
from diffusers import EulerAncestralDiscreteScheduler
from diffusers.utils import load_image, make_image_grid
from huggingface_hub import hf_hub_download
from collage_diffusion import CollageDiffusionPipeline, CollageLayer, compose_collage_layers
torch.set_num_threads(os.cpu_count() or 1)
print(f"CPU threads: {torch.get_num_threads()}")
base = "sd2-community/stable-diffusion-2-1-base"
base_revision = "4e63672c03103b6c636b8fb4119ba982469b2955"
asset_repo_id = "creative-graphic-design/collage-diffusion"
asset_revision = "3d958ad645d9d4641126c8d8a3e74116f4a99421"
asset_root = "source-assets"
asset_url = "https://huggingface.co/creative-graphic-design/collage-diffusion/resolve/3d958ad645d9d4641126c8d8a3e74116f4a99421/source-assets"
scenes = [
{
"name": "Toy Box",
"folder": "toyBox2",
"output": "toy-box-diffusers-comparison-99b4a70.png",
"prompt": "a teddy bear, a wood train, and an american football, in front of a tan background",
"negative_prompt": "collage",
"strength": 0.8,
"noise_blur": 20,
"layers": [
("_0004_background.png", "background", 0.8, 0.0, 0.5, None, None),
("_0003_football.png", "american football", 0.2, 0.5, 0.5, None, None),
("_0002_teddy_bear.png", "teddy bear", 0.8, 0.2, 0.5, "<fda>", "asset_library/toyBox2-teddy_bear/lora_weight_e999_s1000.ti.pt"),
("_0000_wood_train.png", "wood train", 0.5, 0.2, 0.5, "<xtc>", "asset_library/toyBox2-wood_train/lora_weight_e999_s1000.ti.pt"),
],
},
{
"name": "Bento Box",
"folder": "bentoBox",
"output": "bento-box-diffusers-comparison-99b4a70.png",
"prompt": "a bento box with rice, edamame, ginger, and sushi",
"negative_prompt": "collage",
"strength": 0.8,
"noise_blur": 20,
"layers": [
("_0004_bento_box.png", "bento box", 0.5, 0.0, 0.5, "<krk>", "asset_library/bentoBox-bento_box/lora_weight_e999_s1000.ti.pt"),
("_0003_rice.png", "rice", 0.5, 0.4, 0.5, "<lol>", "asset_library/bentoBox-rice/lora_weight_e499_s500.ti.pt"),
("_0002_edamame.png", "edamame", 0.6, 0.0, 0.5, "<fda>", "asset_library/bentoBox-edamame/lora_weight_e999_s1000.ti.pt"),
("_0001_ginger.png", "ginger", 0.8, 0.6, 0.5, "<xtc>", "asset_library/bentoBox-ginger/lora_weight_e999_s1000.ti.pt"),
("_0000_sushi.png", "sushi", 0.8, 0.2, 0.5, "<kyt>", "asset_library/bentoBox-sushi/lora_weight_e999_s1000.ti.pt"),
],
},
{
"name": "Cake Table",
"folder": "cakeTable2",
"output": "cake-table-diffusers-comparison-99b4a70.png",
"prompt": "a wood table with two white chairs behind, two pink cakes on top, maroon bookshelves behind, and winter window",
"negative_prompt": "collage, warped, cupcake",
"strength": 0.75,
"noise_blur": 10,
"layers": [
("_0006_winter_window.png", "winter window", 0.8, 0.45, 1.0, "<krk>", "asset_library/cakeTable2-winter_window/lora_weight_e499_s500.ti.pt"),
("_0005_maroon_bookshelves.png", "maroon bookshelves", 0.8, 0.35, 1.0, "<lol>", "asset_library/cakeTable2-maroon_bookshelves/lora_weight_e999_s1000.ti.pt"),
("_0004_white_chairs.png", "white chairs", 0.8, 0.5, 1.0, "<fda>", "asset_library/cakeTable2-white_chairs/lora_weight_e999_s1000.ti.pt"),
("_0003_wood_table.png", "wood table", 0.6, 0.1, 1.0, "<xtc>", "asset_library/cakeTable2-wood_table/lora_weight_e999_s1000.ti.pt"),
("_0002_decorated_cakes.png", "pink cakes", 0.6, 0.3, 2.0, "<kyt>", "asset_library/cakeTable2-decorated_cakes/lora_weight_e99_s100.ti.pt"),
],
},
]
selected_names = os.environ.get("COLLAGE_SCENES")
if selected_names:
selected_names = {name.strip() for name in selected_names.split(",")}
scenes = [scene for scene in scenes if scene["name"] in selected_names]
if not scenes:
raise ValueError("COLLAGE_SCENES must name at least one configured scene")
for scene in scenes:
scene_started = time.perf_counter()
layers = []
textual_inversion_embeddings = {}
for filename, prompt, noise, positive, negative, token, embedding in scene["layers"]:
layers.append(
CollageLayer(
image=load_image(
f"{asset_url}/image_layers/{scene['folder']}/layers/{filename}",
convert_method=lambda image: image.convert("RGBA"),
),
prompt=prompt,
noise_level=noise,
positive_attention_weight=positive,
negative_attention_weight=negative,
textual_inversion_token=token,
)
)
if token is not None and embedding is not None:
textual_inversion_embeddings[token] = hf_hub_download(
repo_id=asset_repo_id,
repo_type="model",
revision=asset_revision,
filename=f"{asset_root}/{embedding}",
)
input_collage, _ = compose_collage_layers(layers, width=512, height=512)
scheduler = EulerAncestralDiscreteScheduler.from_pretrained(
base, subfolder="scheduler", revision=base_revision
)
pipe = CollageDiffusionPipeline.from_pretrained(
base, revision=base_revision, scheduler=scheduler
)
result = pipe(
prompt=scene["prompt"],
layers=layers,
textual_inversion_embeddings=textual_inversion_embeddings,
negative_prompt=scene["negative_prompt"],
strength=scene["strength"],
num_inference_steps=50,
guidance_scale=7.5,
noise_blur=scene["noise_blur"],
seed=0,
).images[0]
make_image_grid([input_collage, result], rows=1, cols=2).save(scene["output"])
elapsed_minutes = (time.perf_counter() - scene_started) / 60
print(f"Saved {scene['output']} in {elapsed_minutes:.1f} minutes")
del pipe, scheduler, result, input_collage, layers, textual_inversion_embeddings
gc.collect()
```
### Downstream Use
Generated images may be used for design exploration after human review of object placement, appearance, and the licenses attached to all input layers and model weights.
### Out-of-Scope Use
This pipeline does not guarantee semantic accuracy, rights clearance for caller-provided images or embeddings, or suitability for safety-critical applications.
## Bias, Risks, and Limitations
Results inherit the capabilities and biases of Stable Diffusion 2.1 and the caller's prompts, layers, masks, and textual-inversion embeddings. The original authors and rights holders of the notebook-linked example images and embeddings, and the license for each file, are unknown. The [source-assets inventory](https://huggingface.co/creative-graphic-design/collage-diffusion/blob/main/source-assets/README.md) records the retrieval URLs and integrity data. The paper's ControlNet extension and textual-inversion training are not implemented here.
### Recommendations
Review each output and confirm rights to every input layer, embedding, and generated asset before publication or commercial use.
## How to Get Started with the Model
Install the pipeline package and `laygen` from commit-pinned GitHub source archives. The repository README uses the `main` archive URL so the command stays current; the published Hub card pins both packages to the commit that published its pipeline code. The archive includes the selected package directory without fetching vendor submodules.
```bash
pip install \
"laygen @ https://github.com/creative-graphic-design/design-generators/archive/987f8df05e59d8e40dfc3f2e0fa941da22314ff4.zip#subdirectory=lib/laygen" \
"collage-diffusion @ https://github.com/creative-graphic-design/design-generators/archive/987f8df05e59d8e40dfc3f2e0fa941da22314ff4.zip#subdirectory=models/collage-diffusion"
```
The Hub pipeline loads code from `creative-graphic-design/collage-diffusion` and weights from `sd2-community/stable-diffusion-2-1-base`. Before setting `trust_remote_code=True`, review the pipeline code in the [Collage Diffusion Hub repository](https://huggingface.co/creative-graphic-design/collage-diffusion); `diffusers` executes that code when it loads the custom pipeline.
```python
from PIL import Image
from diffusers import DiffusionPipeline
from collage_diffusion import CollageLayer
base = "sd2-community/stable-diffusion-2-1-base"
base_revision = "4e63672c03103b6c636b8fb4119ba982469b2955"
pipe = DiffusionPipeline.from_pretrained(
base,
revision=base_revision,
custom_pipeline="creative-graphic-design/collage-diffusion",
trust_remote_code=True,
)
layer = CollageLayer(
image=Image.open("toy.png").convert("RGBA"),
prompt="a small wooden toy",
position=(64, 96),
noise_level=0.35,
positive_attention_weight=0.5,
negative_attention_weight=0.5,
)
result = pipe(
prompt="a small wooden toy on a studio table",
layers=[layer],
negative_prompt="collage",
seed=0,
num_inference_steps=20,
)
result.images[0].save("harmonized.png")
print(type(pipe).__name__)
```
```text
CollageDiffusionPipeline
```
The Hub card pins both source archives to the repository commit that published the pipeline. This package has no learned checkpoints; the pipeline reads base weights from the upstream Stable Diffusion repository and accepts caller-provided layers and optional textual-inversion embeddings.
```bash
git clone https://github.com/creative-graphic-design/design-generators.git
cd design-generators
uv sync --package collage-diffusion
uv run --package collage-diffusion python
```
```python
from diffusers import EulerAncestralDiscreteScheduler
from collage_diffusion import CollageDiffusionPipeline
base = "sd2-community/stable-diffusion-2-1-base"
revision = "4e63672c03103b6c636b8fb4119ba982469b2955"
# Cross-attention guidance uses the scheduler's sigma values.
scheduler = EulerAncestralDiscreteScheduler.from_pretrained(
base, subfolder="scheduler", revision=revision
)
pipe = CollageDiffusionPipeline.from_pretrained(
base, revision=revision, scheduler=scheduler
)
print(type(pipe).__name__)
```
## Training Details
### Training Data
No training dataset is packaged. Layer images and optional textual-inversion embeddings are caller inputs.
### Training Procedure
This package contains no training procedure and no Collage Diffusion checkpoint.
#### Training Hyperparameters
- **Training regime:** Not applicable; the pipeline uses an upstream inference checkpoint.
## Evaluation
### Testing Data, Factors & Metrics
#### Testing Data
Ordinary tests use small synthetic RGBA layers. The comparison uses the Bento Box, Cake Table, and Toy Box scenes and their notebook-linked embeddings; public archive copies are linked above.
#### Factors
The reference comparison holds the input layers, preprocessing, textual-inversion embedding, scheduler, seed, and number of denoising steps constant between the notebook path and this package.
#### Metrics
Compare normalized final RGB arrays with RMSE, maximum absolute difference, and PSNR, and compare each post-scheduler, post-preservation latent with RMSE. The six comparisons use three 512 × 512 scenes and seeds 0 and 1, the same pinned checkpoint, each scene's layers and embeddings, Euler Ancestral, and 10 requested steps (7 or 8 active steps according to strength). Bento Box and Toy Box use strength `0.8` with `noise_blur=20`; Cake Table uses strength `0.75` with `noise_blur=10`.
The notebook example omits `num_inference_steps`, so it uses the pipeline default of 50 steps (40 active at strength `0.8` and 37 at `0.75`). The parity run requests 10 steps to limit comparison cost (8 or 7 active); the notebook and package use identical settings within each comparison pair.
### Parity Results
An independent acceptance rerun passed all six scene-and-seed conditions against the authors' notebook. This is tolerance-verified agreement, not bitwise equality. The reference uses `diffusers` `0.10.0`, Torch `2.0.0+cpu`, torchvision `0.15.1+cpu`, and [`🤗transformers`](https://huggingface.co/docs/transformers/index) `4.28.1`; the package uses `diffusers` `0.39.0`, Torch `2.13.0+cpu`, torchvision `0.28.0+cpu`, and `transformers` `5.13.0`. Both comparison environments used CPU with one thread. The ordinary Stable Diffusion Img2Img controls and comparison plan are recorded in [issue #484](https://github.com/creative-graphic-design/design-generators/issues/484).
Each threshold uses `T = 2·Bmax − Bmedian` over the six ordinary Stable Diffusion controls: RGB RMSE has `Bmax=6.00713e-5`, `Bmedian=1.61880e-5`, and `T=1.03955e-4`; maximum per-step latent RMSE has `Bmax=1.01644e-4`, `Bmedian=4.42415e-5`, and `T=1.59046e-4`.
Each RMSE is compared with zero using `rtol=0`; the RGB criterion uses `atol=1.03955e-4` and the latent criterion uses `atol=1.59046e-4`.
The implementation-run measurements below are recorded in this [issue #484 comment](https://github.com/creative-graphic-design/design-generators/issues/484#issuecomment-6636816601). Each row reports the final normalized RGB RMSE, PSNR (peak 1.0), maximum absolute RGB difference, maximum per-step post-scheduler/post-preservation latent RMSE, and first U-Net output difference (maximum absolute difference and RMSE).
| Scene | Seed | Active steps | RGB RMSE | PSNR (dB) | RGB max abs | Max per-step latent RMSE | First U-Net max abs | First U-Net RMSE |
| ---------- | ---: | -----------: | -----------------: | --------------: | -----------------: | -----------------------: | ------------------: | -----------------: |
| Bento Box | 0 | 8 | `3.00031813940e-5` | `90.4566538465` | `4.99486923218e-4` | `6.88358887783e-5` | `2.07424163818e-5` | `6.67105892839e-7` |
| Bento Box | 1 | 8 | `5.48811255684e-6` | `105.211539803` | `8.64267349243e-5` | `5.32430305242e-6` | `1.99079513550e-5` | `6.69578823937e-7` |
| Cake Table | 0 | 7 | `1.26997889517e-5` | `97.9240699240` | `3.05831432343e-4` | `1.02028460064e-5` | `9.88245010376e-5` | `1.79971479210e-6` |
| Cake Table | 1 | 7 | `5.10831144860e-6` | `105.834452642` | `6.95586204529e-5` | `7.19363812652e-6` | `9.45329666138e-5` | `1.75942494680e-6` |
| Toy Box | 0 | 8 | `1.04906613643e-5` | `99.5839426331` | `1.33097171783e-4` | `7.98777295022e-6` | `3.38554382324e-5` | `9.17125873657e-7` |
| Toy Box | 1 | 8 | `1.17187871964e-5` | `98.6223466421` | `4.33802604675e-4` | `1.76268145823e-5` | `3.25739383698e-5` | `9.24976044259e-7` |
The independent reviewer measured these per-condition results. Values are reported as RGB RMSE / maximum per-step latent RMSE; each is below the fixed RGB and latent thresholds above.
| Scene | Seed | RGB RMSE | Maximum per-step latent RMSE |
| ---------- | ---: | -----------------: | ---------------------------: |
| Bento Box | 0 | `9.85132492492e-6` | `9.11307774482e-6` |
| Bento Box | 1 | `1.78408690247e-5` | `5.76063468136e-6` |
| Cake Table | 0 | `1.43160028885e-5` | `7.43683456343e-6` |
| Cake Table | 1 | `1.20999262219e-5` | `7.7582077503e-6` |
| Toy Box | 0 | `2.03771846015e-5` | `1.37336492006e-5` |
| Toy Box | 1 | `1.27287351725e-5` | `1.90641440735e-5` |
The Cake Table checkpoint-step control kept the notebook's per-layer inversion steps `[500, 1000, 1000, 1000, 100]` in layer order. Setting all five to `1000` changed the seed-0 RGB RMSE to `0.120`, showing that the selected checkpoint step for each layer affects the result.
## Reproducibility
See [REPRODUCING.md](https://github.com/creative-graphic-design/design-generators/blob/main/models/collage-diffusion/REPRODUCING.md) for CPU test and notebook-comparison steps.
## Environmental Impact
No model training is performed. CPU validation cost depends on the selected Colab memory profile, dependency installation, and comparison resolution.
## Technical Specifications
### Model Architecture and Objective
The pipeline uses Stable Diffusion image-to-image components with a `diffusers` attention processor. It composites layers in back-to-front order, derives visible alpha masks, adjusts cross-attention logits for layer tokens, and blends the initial latent into each denoising step according to the inverted per-layer noise map.
### Compute Infrastructure
CPU is sufficient for small contract tests. Image generation uses the upstream Stable Diffusion checkpoint and is substantially slower on CPU than on a GPU.
#### Hardware
The requested implementation checks and image comparison run on a named Colab CPU session. No GPU is required for the planned comparison.
#### Software
Use Python 3.11 or newer, `diffusers`, `transformers`, PyTorch, Pillow, NumPy, and the repository's `laygen` workspace library. The pipeline uses the same workspace dependency configuration as other model packages.
## License
Package code follows the repository's Apache-2.0 license. The upstream notebook and UI source repositories do not declare a repository license; their implementations are referenced but not copied or distributed. Stable Diffusion 2.1 base is subject to its upstream CreativeML Open RAIL++-M terms. The original authors and rights holders of the notebook-linked example images and embeddings, and the license applicable to each file, are unknown. The archive records the URL from which each file was retrieved and its SHA-256; these records identify retrieval locations, not rights holders or license grants.
## Citation
```bibtex
@article{sarukkai2023collagediffusion,
title = {Collage Diffusion},
author = {Sarukkai, Vishnu and Li, Linden and Ma, Arden and Ré, Christopher and Fatahalian, Kayvon},
journal = {arXiv preprint arXiv:2303.00262},
year = {2023},
eprint = {2303.00262},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = "https://arxiv.org/abs/2303.00262",
}
```
<!-- --8<-- [end:card] -->
|