| | --- |
| | license: openrail |
| | datasets: |
| | - ChristophSchuhmann/LAION-5B-EN-Aesthetics-Subset_above_6 |
| | --- |
| | |
| |  |
| |
|
| | Image Mixer is a model that lets you combine the concepts, styles, and compositions from multiple images (and text prompts too) and generate new images. |
| |
|
| | It was trained by [Justin Pinkney](https://www.justinpinkney.com) at [Lambda Labs](https://lambdalabs.com/). |
| |
|
| | ## Training details |
| |
|
| | This model is a fine tuned version of [Stable Diffusion Image Variations](https://huggingface.co/lambdalabs/sd-image-variations-diffusers) |
| | it has been trained to accept multiple CLIP embedding concatenated along the sequence dimension (as opposed to 1 in the original model). |
| | During training up to 5 crops of the training images are taken and CLIP embeddings are extracted, these are concatenated and used as the conditioning for the model. |
| | At inference time, CLIP embeddings from multiple images can be used to generate images which are influence by multiple inputs. |
| |
|
| | Training was done at 640x640 on a subset of LAION improved aesthetics, using 8xA100 from [Lambda GPU Cloud](https://cloud.lambdalabs.com). |
| |
|
| | _Note text captions were not used during training of the model, |
| | although input text embeddings works to some extent during inference, the model is primarily designed to accept image embeddings_ |
| |
|
| | ## Usage |
| |
|
| | The model is available on [huggingface spaces](https://huggingface.co/spaces/lambdalabs/image-mixer-demo) or to run locally do the following: |
| |
|
| | ```bash |
| | git clone https://github.com/justinpinkney/stable-diffusion.git |
| | cd stable-diffusion |
| | git checkout 1c8a598f312e54f614d1b9675db0e66382f7e23c |
| | python -m venv .venv --prompt sd |
| | . .venv/bin/activate |
| | pip install -U pip |
| | pip install -r requirements.txt |
| | python scripts/gradio_image_mixer.py |
| | ``` |
| |
|
| | Then navigate to the gradio demo link printed in the terminal. |
| |
|
| | For details on how to use the model outside the app refer to the [`run` function](https://github.com/justinpinkney/stable-diffusion/blob/c1963a36a4f8ce23784c8247fa1af0e34e02b766/scripts/gradio_image_mixer.py#L79) in `gradio_image_mixer.py` in the [original repo](https://github.com/justinpinkney/stable-diffusion#image-mixer) |
