README.md

---
library_name: diffusers
pipeline_tag: image-to-text
base_model: google/gemma-4-E4B-it
tags:
  - image-to-text
  - captioning
  - modular-diffusers
  - ideogram
license: apache-2.0
---

# Ideogram4 caption block

A custom [Modular Diffusers](https://huggingface.co/docs/diffusers/main/en/modular_diffusers/overview) block that
produces **Ideogram 4's native structured-JSON caption** from either an image or a short text idea, using a Gemma-4
vision-language model. The caption it returns can be fed straight into the Ideogram 4 generation pipeline as `prompt`.

```
image  -> caption it -> { high_level_description, style_description, compositional_deconstruction }
prompt -> enhance -> { high_level_description, compositional_deconstruction }
```

The mode is chosen automatically: pass `image` to caption it, or `prompt` (with no image) to enhance a short idea.
Prompt enhancement reuses Ideogram's canonical magic-prompt system message from
`diffusers.pipelines.ideogram4.prompt_enhancer`.

## Loading & running

```python
import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image

pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_caption_blocks", trust_remote_code=True)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")

image = load_image("your_image.png")
caption = pipe(image=image, output="caption")   # caption the image -> JSON string
print(caption)
```

Enhance a text idea instead (no image) — Ideogram's "magic prompt":

```python
caption = pipe(
    prompt="a cozy coffee shop on a rainy evening",
    output="caption",
)
```

Get the parsed dict instead (or alongside):

```python
out = pipe(image=image, output=["caption", "caption_json"])
out["caption_json"]   # dict, or None if the model output couldn't be parsed
```

Inputs: `image` (caption it) **or** `prompt` (enhance it); `instruction` (image-mode schema prompt), `height`/`width`
(aspect-ratio hint for enhance mode), `max_new_tokens` (2048), `temperature` (0.0 = greedy). Outputs: `caption`
(pretty JSON string), `caption_json` (parsed dict), `caption_raw` (raw decoded text).

## Caption → generate

```python
caption = pipe(image=ref, output="caption")
# feed straight into the Ideogram 4 generation pipeline (see OzzyGT/ideogram4-modular)
image = gen_pipe(prompt=caption, output="images")[0]
```

## Notes

- **Default checkpoint:** `google/gemma-4-E4B-it` (official bf16, ~15 GB — gated, needs a license-accepted HF
  token and a >=24 GB GPU). Requires `transformers>=5.12`. To run on a smaller GPU, point
  `pretrained_model_name_or_path` at a quantized checkpoint of the same model — import its quantization backend
  first, as the block no longer bundles one.
- Ideogram 4 was trained on these JSON captions, so a caption from this block is the ideal `prompt` for
  re-generation / auto-captioned img2img.