--- language: - en library_name: "diffusers" pipeline_tag: "image-to-image" tags: - "collage-diffusion" - "image-generation" - "wacv-2024" datasets: [] base_model: "sd2-community/stable-diffusion-2-1-base" --- # Model Card for Collage Diffusion [![arXiv](https://img.shields.io/static/v1?label=arXiv&message=2303.00262&color=b31b1b&style=flat-square&logo=arxiv&logoColor=white)](https://arxiv.org/abs/2303.00262) ![venue](https://img.shields.io/static/v1?label=venue&message=WACV+2024&color=purple&style=flat-square) ![license](https://img.shields.io/static/v1?label=license&message=Apache-2.0&color=green&style=flat-square&logo=apache&logoColor=white) ![framework](https://img.shields.io/static/v1?label=framework&message=diffusers&color=red&style=flat-square&logo=huggingface&logoColor=white) ![base](https://img.shields.io/static/v1?label=base&message=diffusers&color=blue&style=flat-square&logo=huggingface&logoColor=white) ![task](https://img.shields.io/static/v1?label=task&message=image-to-image&color=purple&style=flat-square) ![vendor-parity](https://img.shields.io/static/v1?label=vendor-parity&message=tolerance-verified&color=success&style=flat-square) ![hub](https://img.shields.io/static/v1?label=hub&message=published-code-only&color=brightgreen&style=flat-square&logo=huggingface&logoColor=white) This package exposes Collage Diffusion as a [`🧨diffusers`](https://huggingface.co/docs/diffusers/index) pipeline that harmonizes text-described RGBA layers into a single image. ## Model Details ### Model Description Collage Diffusion combines a scene prompt with ordered RGBA image layers. Visible alpha masks guide cross-attention to each layer's text, while per-layer noise levels control how much of each source layer is preserved during diffusion. The pipeline can load caller-provided textual-inversion embeddings. It uses the upstream Stable Diffusion 2.1 base checkpoint and does not train, store, or publish model-specific weights. - **Developed by:** Vishnu Sarukkai, Linden Li, Arden Ma, Christopher Ré, and Kayvon Fatahalian. - **Shared by:** creative-graphic-design. - **Model type:** image generation; task: image-to-image; conditioning: layers, text. - **Language(s) (NLP):** English scene and layer prompts. - **License:** Apache-2.0 for this package; the base checkpoint and input assets retain their own terms. - **Finetuned from:** No Collage Diffusion checkpoint; inference loads `sd2-community/stable-diffusion-2-1-base` at the pinned revision below under the CreativeML Open RAIL++-M terms declared by its model card. ### Model Sources - **Repository:** [Collage Diffusion source repository](https://github.com/VSAnimator/collage-diffusion); it has no declared repository license. Its notebook is used as a behavioral reference; its code is not included here. Notebook-linked image layers and textual-inversion embeddings are archived, with source links and integrity records, in the [model repository's source-assets directory](https://huggingface.co/creative-graphic-design/collage-diffusion/tree/main/source-assets). - **Paper:** [Collage Diffusion](https://arxiv.org/abs/2303.00262). - **Related implementation:** [Collage Diffusion UI](https://github.com/linden-li/collage-diffusion-ui) was inspected as a secondary reference. Its repository also has no declared license; its code is not included here. - **Released weights:** The package does not contain its own weights. It loads the [Stable Diffusion 2.1 base mirror](https://huggingface.co/sd2-community/stable-diffusion-2-1-base) at a pinned revision. ### Base Checkpoint Provenance and Integrity The authors' notebook names `stabilityai/stable-diffusion-2-1-base`, but that Hub repository now returns HTTP 404. The comparison and package examples use `sd2-community/stable-diffusion-2-1-base` at revision `4e63672c03103b6c636b8fb4119ba982469b2955`. The model card declares the CreativeML Open RAIL++-M license; users must review and follow those terms. The main weight files were checked across three available copies. Their SHA-256 values match byte-for-byte: | Copy | Revision | UNet SHA-256 | VAE SHA-256 | Text encoder SHA-256 | | -------------------------------------------- | ------------------------------------------ | ------------------------------------------------------------------ | ------------------------------------------------------------------ | ------------------------------------------------------------------ | | `sd2-community/stable-diffusion-2-1-base` | `4e63672c03103b6c636b8fb4119ba982469b2955` | `6dfae3e5f7d459b50f4b0850ead945972c75bb0e1897628933e169eb43974214` | `a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815` | `cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c` | | `Manojb/stable-diffusion-2-1-base` | `0094d483a120f3f33dafbd187ea4aa60d10de75c` | same | same | same | | `LanguageMachines/stable-diffusion-2-1-base` | `2579f99c90b5a3019fc115ee9566d5ec4f17fccf` | same | same | same | SHA-256 values for every file used from the selected pinned snapshot: | File | SHA-256 | | -------------------------------------------- | ------------------------------------------------------------------ | | `model_index.json` | `518553e098fdaa4153d9a92139e344e44f53bb329c22f383293e3099c4f1bad2` | | `feature_extractor/preprocessor_config.json` | `2a1da83b5e1032aaeef397552ddb408dca0d8cd1dc58f61bf6abf38d6f33a0a2` | | `scheduler/scheduler_config.json` | `11ac5627d7df0fa344b875c4b5722b1767a8a2aa1684c2cf8b4d614300127234` | | `text_encoder/config.json` | `7e33bfc475f6cf76e69b28b101993ca834d1f22ff4d35caefde32d63d4f42fba` | | `text_encoder/model.safetensors` | `cce6febb0b6d876ee5eb24af35e27e764eb4f9b1d0b7c026c8c3333d4cfc916c` | | `tokenizer/merges.txt` | `9fd691f7c8039210e0fced15865466c65820d09b63988b0174bfe25de299051a` | | `tokenizer/special_tokens_map.json` | `f118ab3a983206e4f32583448de6bd6aae4ee21869135cef1f5848a753cdaab6` | | `tokenizer/tokenizer_config.json` | `d562b18a0c6d32a168fce3d7a342fce64579ebacffb98fb71b0affc494fc32d8` | | `tokenizer/vocab.json` | `e089ad92ba36837a0d31433e555c8f45fe601ab5c221d4f607ded32d9f7a4349` | | `unet/config.json` | `ce0c6d379e3b1d3e1f79338de70c80a1e36f17a6372439a8249b6fb0dfa1b608` | | `unet/diffusion_pytorch_model.safetensors` | `6dfae3e5f7d459b50f4b0850ead945972c75bb0e1897628933e169eb43974214` | | `vae/config.json` | `424117cb534ce03497c41305ed868980123917b2b6abba4bbaa615e968772903` | | `vae/diffusion_pytorch_model.safetensors` | `a1d993488569e928462932c8c38a0760b874d166399b14414135bd9c42df5815` | Both sides of the notebook comparison use this same pinned snapshot and the file hashes listed above. The selected model copy and its weights are not redistributed by this package. ## Supported Checkpoints | Artifact | Hub ID | Status | | ------------------------- | ------------------------------------------- | ----------------------------------------- | | `diffusers` pipeline code | `creative-graphic-design/collage-diffusion` | planned, code only; no learned checkpoint | When publishing this pipeline to the Hub, its `model_index.json` and `scheduler/scheduler_config.json` must specify `EulerAncestralDiscreteScheduler`. ## Uses ### Direct Use Use the pipeline to harmonize a layered collage while guiding object placement through masks and controlling source preservation through per-layer noise levels. ```python from PIL import Image from diffusers import EulerAncestralDiscreteScheduler from collage_diffusion import CollageDiffusionPipeline, CollageLayer base = "sd2-community/stable-diffusion-2-1-base" revision = "4e63672c03103b6c636b8fb4119ba982469b2955" # Cross-attention guidance uses the scheduler's sigma values. scheduler = EulerAncestralDiscreteScheduler.from_pretrained( base, subfolder="scheduler", revision=revision ) pipe = CollageDiffusionPipeline.from_pretrained( base, revision=revision, scheduler=scheduler ) layers = [ CollageLayer( image=Image.open("toy.png").convert("RGBA"), prompt="a small wooden toy", position=(64, 96), noise_level=0.35, positive_attention_weight=0.5, negative_attention_weight=0.5, ), ] result = pipe( prompt="a small wooden toy on a studio table", layers=layers, negative_prompt="collage", seed=0, num_inference_steps=20, ) result.images[0].save("harmonized.png") print("Saved harmonized.png") ``` ```text Saved harmonized.png ``` Textual-inversion embeddings are caller-supplied with a token-to-file mapping, and the matching token is set on its `CollageLayer`. ControlNet and textual-inversion training are outside this package's current scope. ### Examples #### Shared setup These examples follow the [Collage Diffusion notebook](https://github.com/VSAnimator/collage-diffusion/blob/main/CollageDiffusionDemo.ipynb) and its [Google Drive image layers](https://drive.google.com/drive/folders/1EfoLfWbn5fl7c092uk96zHtJOVHRlIqI?usp=sharing) and [textual-inversion assets](https://drive.google.com/drive/folders/13Gg8y-0FxdPcuhrPOUw6Ze9wa0Ruu-cI?usp=sharing). Because the Drive files may disappear, copies and their source URLs, retrieval dates, sizes, and SHA-256 values are recorded in the [model repository's source-assets directory](https://huggingface.co/creative-graphic-design/collage-diffusion/tree/main/source-assets). The original author, rights holder, and per-file license are unknown; the documentation-images dataset's Apache-2.0 license does not apply to those files. Each image places the input collage on the left and the generated result on the right. All scenes use `sd2-community/stable-diffusion-2-1-base` at revision `4e63672c03103b6c636b8fb4119ba982469b2955`, `EulerAncestralDiscreteScheduler`, seed `0`, 512 × 512 resolution, guidance scale `7.5`, and 50 requested inference steps. The active step count is 40 at strength `0.8` and 37 at `0.75`. The base checkpoint uses the [CreativeML Open RAIL++-M License](https://huggingface.co/sd2-community/stable-diffusion-2-1-base). The code uses every CPU thread available to its runtime; the generated examples used **2 threads**. #### Toy Box Prompt: `a teddy bear, a wood train, and an american football, in front of a tan background`. Negative prompt: `collage`. Strength: `0.8`; `noise_blur=20`; 50 requested (40 active) steps. ![Toy Box input collage on the left and Collage Diffusion output on the right.](https://huggingface.co/datasets/creative-graphic-design/documentation-images/resolve/main/collage-diffusion/toy-box-diffusers-comparison-99b4a70.png) #### Bento Box Prompt: `a bento box with rice, edamame, ginger, and sushi`. Negative prompt: `collage`. Strength: `0.8`; `noise_blur=20`; 50 requested (40 active) steps. ![Bento Box input collage on the left and Collage Diffusion output on the right.](https://huggingface.co/datasets/creative-graphic-design/documentation-images/resolve/main/collage-diffusion/bento-box-diffusers-comparison-99b4a70.png) #### Cake Table Prompt: `a wood table with two white chairs behind, two pink cakes on top, maroon bookshelves behind, and winter window`. Negative prompt: `collage, warped, cupcake`. Strength: `0.75`; `noise_blur=10`; 50 requested (37 active) steps. ![Cake Table input collage on the left and Collage Diffusion output on the right.](https://huggingface.co/datasets/creative-graphic-design/documentation-images/resolve/main/collage-diffusion/cake-table-diffusers-comparison-99b4a70.png) The following script builds each scene from its archived image layers and textual-inversion embeddings, then saves the side-by-side examples above. It generates all three by default; set `COLLAGE_SCENES` to a comma-separated subset of scene names to run only selected examples. ```python import gc import os import time import torch from diffusers import EulerAncestralDiscreteScheduler from diffusers.utils import load_image, make_image_grid from huggingface_hub import hf_hub_download from collage_diffusion import CollageDiffusionPipeline, CollageLayer, compose_collage_layers torch.set_num_threads(os.cpu_count() or 1) print(f"CPU threads: {torch.get_num_threads()}") base = "sd2-community/stable-diffusion-2-1-base" base_revision = "4e63672c03103b6c636b8fb4119ba982469b2955" asset_repo_id = "creative-graphic-design/collage-diffusion" asset_revision = "3d958ad645d9d4641126c8d8a3e74116f4a99421" asset_root = "source-assets" asset_url = "https://huggingface.co/creative-graphic-design/collage-diffusion/resolve/3d958ad645d9d4641126c8d8a3e74116f4a99421/source-assets" scenes = [ { "name": "Toy Box", "folder": "toyBox2", "output": "toy-box-diffusers-comparison-99b4a70.png", "prompt": "a teddy bear, a wood train, and an american football, in front of a tan background", "negative_prompt": "collage", "strength": 0.8, "noise_blur": 20, "layers": [ ("_0004_background.png", "background", 0.8, 0.0, 0.5, None, None), ("_0003_football.png", "american football", 0.2, 0.5, 0.5, None, None), ("_0002_teddy_bear.png", "teddy bear", 0.8, 0.2, 0.5, "", "asset_library/toyBox2-teddy_bear/lora_weight_e999_s1000.ti.pt"), ("_0000_wood_train.png", "wood train", 0.5, 0.2, 0.5, "", "asset_library/toyBox2-wood_train/lora_weight_e999_s1000.ti.pt"), ], }, { "name": "Bento Box", "folder": "bentoBox", "output": "bento-box-diffusers-comparison-99b4a70.png", "prompt": "a bento box with rice, edamame, ginger, and sushi", "negative_prompt": "collage", "strength": 0.8, "noise_blur": 20, "layers": [ ("_0004_bento_box.png", "bento box", 0.5, 0.0, 0.5, "", "asset_library/bentoBox-bento_box/lora_weight_e999_s1000.ti.pt"), ("_0003_rice.png", "rice", 0.5, 0.4, 0.5, "", "asset_library/bentoBox-rice/lora_weight_e499_s500.ti.pt"), ("_0002_edamame.png", "edamame", 0.6, 0.0, 0.5, "", "asset_library/bentoBox-edamame/lora_weight_e999_s1000.ti.pt"), ("_0001_ginger.png", "ginger", 0.8, 0.6, 0.5, "", "asset_library/bentoBox-ginger/lora_weight_e999_s1000.ti.pt"), ("_0000_sushi.png", "sushi", 0.8, 0.2, 0.5, "", "asset_library/bentoBox-sushi/lora_weight_e999_s1000.ti.pt"), ], }, { "name": "Cake Table", "folder": "cakeTable2", "output": "cake-table-diffusers-comparison-99b4a70.png", "prompt": "a wood table with two white chairs behind, two pink cakes on top, maroon bookshelves behind, and winter window", "negative_prompt": "collage, warped, cupcake", "strength": 0.75, "noise_blur": 10, "layers": [ ("_0006_winter_window.png", "winter window", 0.8, 0.45, 1.0, "", "asset_library/cakeTable2-winter_window/lora_weight_e499_s500.ti.pt"), ("_0005_maroon_bookshelves.png", "maroon bookshelves", 0.8, 0.35, 1.0, "", "asset_library/cakeTable2-maroon_bookshelves/lora_weight_e999_s1000.ti.pt"), ("_0004_white_chairs.png", "white chairs", 0.8, 0.5, 1.0, "", "asset_library/cakeTable2-white_chairs/lora_weight_e999_s1000.ti.pt"), ("_0003_wood_table.png", "wood table", 0.6, 0.1, 1.0, "", "asset_library/cakeTable2-wood_table/lora_weight_e999_s1000.ti.pt"), ("_0002_decorated_cakes.png", "pink cakes", 0.6, 0.3, 2.0, "", "asset_library/cakeTable2-decorated_cakes/lora_weight_e99_s100.ti.pt"), ], }, ] selected_names = os.environ.get("COLLAGE_SCENES") if selected_names: selected_names = {name.strip() for name in selected_names.split(",")} scenes = [scene for scene in scenes if scene["name"] in selected_names] if not scenes: raise ValueError("COLLAGE_SCENES must name at least one configured scene") for scene in scenes: scene_started = time.perf_counter() layers = [] textual_inversion_embeddings = {} for filename, prompt, noise, positive, negative, token, embedding in scene["layers"]: layers.append( CollageLayer( image=load_image( f"{asset_url}/image_layers/{scene['folder']}/layers/{filename}", convert_method=lambda image: image.convert("RGBA"), ), prompt=prompt, noise_level=noise, positive_attention_weight=positive, negative_attention_weight=negative, textual_inversion_token=token, ) ) if token is not None and embedding is not None: textual_inversion_embeddings[token] = hf_hub_download( repo_id=asset_repo_id, repo_type="model", revision=asset_revision, filename=f"{asset_root}/{embedding}", ) input_collage, _ = compose_collage_layers(layers, width=512, height=512) scheduler = EulerAncestralDiscreteScheduler.from_pretrained( base, subfolder="scheduler", revision=base_revision ) pipe = CollageDiffusionPipeline.from_pretrained( base, revision=base_revision, scheduler=scheduler ) result = pipe( prompt=scene["prompt"], layers=layers, textual_inversion_embeddings=textual_inversion_embeddings, negative_prompt=scene["negative_prompt"], strength=scene["strength"], num_inference_steps=50, guidance_scale=7.5, noise_blur=scene["noise_blur"], seed=0, ).images[0] make_image_grid([input_collage, result], rows=1, cols=2).save(scene["output"]) elapsed_minutes = (time.perf_counter() - scene_started) / 60 print(f"Saved {scene['output']} in {elapsed_minutes:.1f} minutes") del pipe, scheduler, result, input_collage, layers, textual_inversion_embeddings gc.collect() ``` ### Downstream Use Generated images may be used for design exploration after human review of object placement, appearance, and the licenses attached to all input layers and model weights. ### Out-of-Scope Use This pipeline does not guarantee semantic accuracy, rights clearance for caller-provided images or embeddings, or suitability for safety-critical applications. ## Bias, Risks, and Limitations Results inherit the capabilities and biases of Stable Diffusion 2.1 and the caller's prompts, layers, masks, and textual-inversion embeddings. The original authors and rights holders of the notebook-linked example images and embeddings, and the license for each file, are unknown. The [source-assets inventory](https://huggingface.co/creative-graphic-design/collage-diffusion/blob/main/source-assets/README.md) records the retrieval URLs and integrity data. The paper's ControlNet extension and textual-inversion training are not implemented here. ### Recommendations Review each output and confirm rights to every input layer, embedding, and generated asset before publication or commercial use. ## How to Get Started with the Model Install the pipeline package and `laygen` from commit-pinned GitHub source archives. The repository README uses the `main` archive URL so the command stays current; the published Hub card pins both packages to the commit that published its pipeline code. The archive includes the selected package directory without fetching vendor submodules. ```bash pip install \ "laygen @ https://github.com/creative-graphic-design/design-generators/archive/987f8df05e59d8e40dfc3f2e0fa941da22314ff4.zip#subdirectory=lib/laygen" \ "collage-diffusion @ https://github.com/creative-graphic-design/design-generators/archive/987f8df05e59d8e40dfc3f2e0fa941da22314ff4.zip#subdirectory=models/collage-diffusion" ``` The Hub pipeline loads code from `creative-graphic-design/collage-diffusion` and weights from `sd2-community/stable-diffusion-2-1-base`. Before setting `trust_remote_code=True`, review the pipeline code in the [Collage Diffusion Hub repository](https://huggingface.co/creative-graphic-design/collage-diffusion); `diffusers` executes that code when it loads the custom pipeline. ```python from PIL import Image from diffusers import DiffusionPipeline from collage_diffusion import CollageLayer base = "sd2-community/stable-diffusion-2-1-base" base_revision = "4e63672c03103b6c636b8fb4119ba982469b2955" pipe = DiffusionPipeline.from_pretrained( base, revision=base_revision, custom_pipeline="creative-graphic-design/collage-diffusion", trust_remote_code=True, ) layer = CollageLayer( image=Image.open("toy.png").convert("RGBA"), prompt="a small wooden toy", position=(64, 96), noise_level=0.35, positive_attention_weight=0.5, negative_attention_weight=0.5, ) result = pipe( prompt="a small wooden toy on a studio table", layers=[layer], negative_prompt="collage", seed=0, num_inference_steps=20, ) result.images[0].save("harmonized.png") print(type(pipe).__name__) ``` ```text CollageDiffusionPipeline ``` The Hub card pins both source archives to the repository commit that published the pipeline. This package has no learned checkpoints; the pipeline reads base weights from the upstream Stable Diffusion repository and accepts caller-provided layers and optional textual-inversion embeddings. ```bash git clone https://github.com/creative-graphic-design/design-generators.git cd design-generators uv sync --package collage-diffusion uv run --package collage-diffusion python ``` ```python from diffusers import EulerAncestralDiscreteScheduler from collage_diffusion import CollageDiffusionPipeline base = "sd2-community/stable-diffusion-2-1-base" revision = "4e63672c03103b6c636b8fb4119ba982469b2955" # Cross-attention guidance uses the scheduler's sigma values. scheduler = EulerAncestralDiscreteScheduler.from_pretrained( base, subfolder="scheduler", revision=revision ) pipe = CollageDiffusionPipeline.from_pretrained( base, revision=revision, scheduler=scheduler ) print(type(pipe).__name__) ``` ## Training Details ### Training Data No training dataset is packaged. Layer images and optional textual-inversion embeddings are caller inputs. ### Training Procedure This package contains no training procedure and no Collage Diffusion checkpoint. #### Training Hyperparameters - **Training regime:** Not applicable; the pipeline uses an upstream inference checkpoint. ## Evaluation ### Testing Data, Factors & Metrics #### Testing Data Ordinary tests use small synthetic RGBA layers. The comparison uses the Bento Box, Cake Table, and Toy Box scenes and their notebook-linked embeddings; public archive copies are linked above. #### Factors The reference comparison holds the input layers, preprocessing, textual-inversion embedding, scheduler, seed, and number of denoising steps constant between the notebook path and this package. #### Metrics Compare normalized final RGB arrays with RMSE, maximum absolute difference, and PSNR, and compare each post-scheduler, post-preservation latent with RMSE. The six comparisons use three 512 × 512 scenes and seeds 0 and 1, the same pinned checkpoint, each scene's layers and embeddings, Euler Ancestral, and 10 requested steps (7 or 8 active steps according to strength). Bento Box and Toy Box use strength `0.8` with `noise_blur=20`; Cake Table uses strength `0.75` with `noise_blur=10`. The notebook example omits `num_inference_steps`, so it uses the pipeline default of 50 steps (40 active at strength `0.8` and 37 at `0.75`). The parity run requests 10 steps to limit comparison cost (8 or 7 active); the notebook and package use identical settings within each comparison pair. ### Parity Results An independent acceptance rerun passed all six scene-and-seed conditions against the authors' notebook. This is tolerance-verified agreement, not bitwise equality. The reference uses `diffusers` `0.10.0`, Torch `2.0.0+cpu`, torchvision `0.15.1+cpu`, and [`🤗transformers`](https://huggingface.co/docs/transformers/index) `4.28.1`; the package uses `diffusers` `0.39.0`, Torch `2.13.0+cpu`, torchvision `0.28.0+cpu`, and `transformers` `5.13.0`. Both comparison environments used CPU with one thread. The ordinary Stable Diffusion Img2Img controls and comparison plan are recorded in [issue #484](https://github.com/creative-graphic-design/design-generators/issues/484). Each threshold uses `T = 2·Bmax − Bmedian` over the six ordinary Stable Diffusion controls: RGB RMSE has `Bmax=6.00713e-5`, `Bmedian=1.61880e-5`, and `T=1.03955e-4`; maximum per-step latent RMSE has `Bmax=1.01644e-4`, `Bmedian=4.42415e-5`, and `T=1.59046e-4`. Each RMSE is compared with zero using `rtol=0`; the RGB criterion uses `atol=1.03955e-4` and the latent criterion uses `atol=1.59046e-4`. The implementation-run measurements below are recorded in this [issue #484 comment](https://github.com/creative-graphic-design/design-generators/issues/484#issuecomment-6636816601). Each row reports the final normalized RGB RMSE, PSNR (peak 1.0), maximum absolute RGB difference, maximum per-step post-scheduler/post-preservation latent RMSE, and first U-Net output difference (maximum absolute difference and RMSE). | Scene | Seed | Active steps | RGB RMSE | PSNR (dB) | RGB max abs | Max per-step latent RMSE | First U-Net max abs | First U-Net RMSE | | ---------- | ---: | -----------: | -----------------: | --------------: | -----------------: | -----------------------: | ------------------: | -----------------: | | Bento Box | 0 | 8 | `3.00031813940e-5` | `90.4566538465` | `4.99486923218e-4` | `6.88358887783e-5` | `2.07424163818e-5` | `6.67105892839e-7` | | Bento Box | 1 | 8 | `5.48811255684e-6` | `105.211539803` | `8.64267349243e-5` | `5.32430305242e-6` | `1.99079513550e-5` | `6.69578823937e-7` | | Cake Table | 0 | 7 | `1.26997889517e-5` | `97.9240699240` | `3.05831432343e-4` | `1.02028460064e-5` | `9.88245010376e-5` | `1.79971479210e-6` | | Cake Table | 1 | 7 | `5.10831144860e-6` | `105.834452642` | `6.95586204529e-5` | `7.19363812652e-6` | `9.45329666138e-5` | `1.75942494680e-6` | | Toy Box | 0 | 8 | `1.04906613643e-5` | `99.5839426331` | `1.33097171783e-4` | `7.98777295022e-6` | `3.38554382324e-5` | `9.17125873657e-7` | | Toy Box | 1 | 8 | `1.17187871964e-5` | `98.6223466421` | `4.33802604675e-4` | `1.76268145823e-5` | `3.25739383698e-5` | `9.24976044259e-7` | The independent reviewer measured these per-condition results. Values are reported as RGB RMSE / maximum per-step latent RMSE; each is below the fixed RGB and latent thresholds above. | Scene | Seed | RGB RMSE | Maximum per-step latent RMSE | | ---------- | ---: | -----------------: | ---------------------------: | | Bento Box | 0 | `9.85132492492e-6` | `9.11307774482e-6` | | Bento Box | 1 | `1.78408690247e-5` | `5.76063468136e-6` | | Cake Table | 0 | `1.43160028885e-5` | `7.43683456343e-6` | | Cake Table | 1 | `1.20999262219e-5` | `7.7582077503e-6` | | Toy Box | 0 | `2.03771846015e-5` | `1.37336492006e-5` | | Toy Box | 1 | `1.27287351725e-5` | `1.90641440735e-5` | The Cake Table checkpoint-step control kept the notebook's per-layer inversion steps `[500, 1000, 1000, 1000, 100]` in layer order. Setting all five to `1000` changed the seed-0 RGB RMSE to `0.120`, showing that the selected checkpoint step for each layer affects the result. ## Reproducibility See [REPRODUCING.md](https://github.com/creative-graphic-design/design-generators/blob/main/models/collage-diffusion/REPRODUCING.md) for CPU test and notebook-comparison steps. ## Environmental Impact No model training is performed. CPU validation cost depends on the selected Colab memory profile, dependency installation, and comparison resolution. ## Technical Specifications ### Model Architecture and Objective The pipeline uses Stable Diffusion image-to-image components with a `diffusers` attention processor. It composites layers in back-to-front order, derives visible alpha masks, adjusts cross-attention logits for layer tokens, and blends the initial latent into each denoising step according to the inverted per-layer noise map. ### Compute Infrastructure CPU is sufficient for small contract tests. Image generation uses the upstream Stable Diffusion checkpoint and is substantially slower on CPU than on a GPU. #### Hardware The requested implementation checks and image comparison run on a named Colab CPU session. No GPU is required for the planned comparison. #### Software Use Python 3.11 or newer, `diffusers`, `transformers`, PyTorch, Pillow, NumPy, and the repository's `laygen` workspace library. The pipeline uses the same workspace dependency configuration as other model packages. ## License Package code follows the repository's Apache-2.0 license. The upstream notebook and UI source repositories do not declare a repository license; their implementations are referenced but not copied or distributed. Stable Diffusion 2.1 base is subject to its upstream CreativeML Open RAIL++-M terms. The original authors and rights holders of the notebook-linked example images and embeddings, and the license applicable to each file, are unknown. The archive records the URL from which each file was retrieved and its SHA-256; these records identify retrieval locations, not rights holders or license grants. ## Citation ```bibtex @article{sarukkai2023collagediffusion, title = {Collage Diffusion}, author = {Sarukkai, Vishnu and Li, Linden and Ma, Arden and Ré, Christopher and Fatahalian, Kayvon}, journal = {arXiv preprint arXiv:2303.00262}, year = {2023}, eprint = {2303.00262}, archivePrefix = {arXiv}, primaryClass = {cs.CV}, url = "https://arxiv.org/abs/2303.00262", } ```