File size: 2,858 Bytes
32de7f4 bdc1856 32de7f4 bdc1856 a78bf56 bdc1856 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 | ---
library_name: diffusers
pipeline_tag: image-to-text
base_model: google/gemma-4-E4B-it
tags:
- image-to-text
- captioning
- modular-diffusers
- ideogram
license: apache-2.0
---
# Ideogram4 caption block
A custom [Modular Diffusers](https://huggingface.co/docs/diffusers/main/en/modular_diffusers/overview) block that
produces **Ideogram 4's native structured-JSON caption** from either an image or a short text idea, using a Gemma-4
vision-language model. The caption it returns can be fed straight into the Ideogram 4 generation pipeline as `prompt`.
```
image -> caption it -> { high_level_description, style_description, compositional_deconstruction }
prompt -> enhance -> { high_level_description, compositional_deconstruction }
```
The mode is chosen automatically: pass `image` to caption it, or `prompt` (with no image) to enhance a short idea.
Prompt enhancement reuses Ideogram's canonical magic-prompt system message from
`diffusers.pipelines.ideogram4.prompt_enhancer`.
## Loading & running
```python
import torch
from diffusers import ModularPipeline
from diffusers.utils import load_image
pipe = ModularPipeline.from_pretrained("OzzyGT/ideogram4_caption_blocks", trust_remote_code=True)
pipe.load_components(torch_dtype=torch.bfloat16)
pipe.to("cuda")
image = load_image("your_image.png")
caption = pipe(image=image, output="caption") # caption the image -> JSON string
print(caption)
```
Enhance a text idea instead (no image) — Ideogram's "magic prompt":
```python
caption = pipe(
prompt="a cozy coffee shop on a rainy evening",
output="caption",
)
```
Get the parsed dict instead (or alongside):
```python
out = pipe(image=image, output=["caption", "caption_json"])
out["caption_json"] # dict, or None if the model output couldn't be parsed
```
Inputs: `image` (caption it) **or** `prompt` (enhance it); `instruction` (image-mode schema prompt), `height`/`width`
(aspect-ratio hint for enhance mode), `max_new_tokens` (2048), `temperature` (0.0 = greedy). Outputs: `caption`
(pretty JSON string), `caption_json` (parsed dict), `caption_raw` (raw decoded text).
## Caption → generate
```python
caption = pipe(image=ref, output="caption")
# feed straight into the Ideogram 4 generation pipeline (see OzzyGT/ideogram4-modular)
image = gen_pipe(prompt=caption, output="images")[0]
```
## Notes
- **Default checkpoint:** `google/gemma-4-E4B-it` (official bf16, ~15 GB — gated, needs a license-accepted HF
token and a >=24 GB GPU). Requires `transformers>=5.12`. To run on a smaller GPU, point
`pretrained_model_name_or_path` at a quantized checkpoint of the same model — import its quantization backend
first, as the block no longer bundles one.
- Ideogram 4 was trained on these JSON captions, so a caption from this block is the ideal `prompt` for
re-generation / auto-captioned img2img.
|