File size: 2,453 Bytes
9443a8d 0b608dd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
library_name: diffusers
pipeline_tag: text-to-image
base_model: ideogram-ai/ideogram-4-nf4-diffusers
tags:
- text-to-image
- image-to-image
- differential-diffusion
- modular-diffusers
- diffusion
- ideogram
license: apache-2.0
---
# Ideogram4 custom modular blocks
Custom [Modular Diffusers](https://huggingface.co/docs/diffusers/main/en/modular_diffusers/overview) blocks that
extend Ideogram4 with **image-to-image** and **Differential Diffusion**, plus a **unified `AutoBlocks`** that folds
text-to-image, img2img, and differential diffusion into a single pipeline, the workflow is chosen automatically
from which inputs you pass.
```
prompt -> text2image
prompt + image -> image2image (optional strength)
prompt + image + diffdiff_map -> differential diffusion
```
## Loading & running
```python
import torch
from diffusers import ModularPipeline
pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4-modular", trust_remote_code=True)
pipe.load_components(
names=["text_encoder", "tokenizer", "transformer", "unconditional_transformer", "vae", "scheduler"],
torch_dtype=torch.bfloat16,
)
pipe.to("cuda")
image = pipe(prompt="a photo of a red apple on a black background", output="images")[0]
image.save("out.png")
```
### image-to-image
Pass an `image` (and optional `strength`) to the same pipe:
```python
from diffusers.utils import load_image
image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png")
result = pipe(
prompt="a photo of a snowy mountain landscape at sunset, dramatic clouds",
image=image,
strength=0.6,
output="images",
)[0]
result.save("img2img.png")
```
### differential diffusion
Add a grayscale change map (`diffdiff_map`) alongside the `image` — darker regions change more, brighter regions stay closer to the reference:
```python
from diffusers.utils import load_image
image = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/20240329211129_4024911930.png")
diffdiff_map = load_image("https://huggingface.co/datasets/OzzyGT/testing-resources/resolve/main/differential/gradient_mask.png")
result = pipe(
prompt="a photo of a snowy mountain landscape at sunset, dramatic clouds",
image=image,
diffdiff_map=diffdiff_map,
output="images",
)[0]
result.save("diffdiff.png")
``` |