Fix no-CFG example for Diffusers 0.39.0

d47b425 verified 2 days ago

7.65 kB

license: other
license_name: ideogram-4-non-commercial
license_link: https://huggingface.co/fal/ideogram-v4-instant/blob/main/LICENSE.md
library_name: diffusers
pipeline_tag: text-to-image
base_model: ideogram-ai/ideogram-4-fp8
base_model_relation: finetune
tags:
  - diffusers
  - safetensors
  - text-to-image
  - image-generation
  - flow-matching
  - ideogram4
  - distillation
  - cfg-distillation
  - no-cfg
  - no-qad
  - instant
  - 8-step
  - bf16
extra_gated_prompt: >-
  By requesting access, you acknowledge the Ideogram Non-Commercial Model
  Agreement linked above.
extra_gated_fields:
  I agree to use this model only as permitted by the Ideogram Non-Commercial Model Agreement: checkbox

Ideogram 4 Instant — by fal

8 steps. One transformer. No runtime CFG.

Ideogram 4 Instant is an eight-step text-to-image checkpoint developed and released by fal, based on ideogram-ai/ideogram-4-fp8. This release contains the BF16 weights from immediately before the quantization-aware distillation (QAD) stage. It combines timestep distillation with a single conditional branch for generation in just eight denoising steps.

Key features

⚡ 8-step inference — the Instant schedule at 1024×1024.
🎯 No runtime CFG — one conditional transformer call per denoising step; no negative branch or CFG blend.
🧠 Pre-QAD BF16 weights — approximately 9.28 billion parameters, captured immediately before QAD.
🧩 Standard Diffusers components — no repository Python code and no trust_remote_code.
📦 Transformer-only release — shared components come from Ideogram AI's public, gated Diffusers repository.

Read how fal built the single-branch, few-step Ideogram 4 serving path in Serving sub-second Ideogram v4 without quality loss.

Hosted API

The production-optimized model is available on fal through ideogram/v4/instant.

The hosted endpoint uses the later QAD-trained, FP4-optimized production weights. This repository intentionally publishes the BF16 checkpoint from before QAD.

Usage

This model expects Ideogram 4's structured JSON caption format. The hosted fal endpoint expands natural-language prompts automatically; local Diffusers inference does not. Expand the prompt with an Ideogram-compatible magic-prompt model first, or provide a complete structured caption like the one below.

The component wiring below uses the official public, gated ideogram-ai/ideogram-4-nf4-diffusers repository. Only its tokenizer, text encoder, VAE, and scheduler are used; neither of its diffusion transformers is loaded. You must accept Ideogram's access gate before downloading the components.

Released Diffusers 0.39.0 still expects an unconditional transformer even for a distilled, single-branch checkpoint. The zero-parameter compatibility module below satisfies that plumbing without loading or running a second diffusion transformer. With guidance_scale=1.0, the stock blend is exactly 1.0 * conditional + 0.0 * dummy_unconditional.

This shim addresses the mandatory-CFG plumbing only. It does not apply fal's native terminal timestep or frequency-table corrections, so stock Diffusers 0.39.0 is not bit-exact with the optimized fal runtime.

import json

import torch
from diffusers import Ideogram4Pipeline, Ideogram4Transformer2DModel

repo_id = "fal/ideogram-v4-instant"
components_repo_id = "ideogram-ai/ideogram-4-nf4-diffusers"
components_revision = "1874bc70267ba2c823a7239e1d70dd308c8d64dc"


class ZeroUnconditionalTransformer(torch.nn.Module):
    """Zero-parameter stand-in for Diffusers 0.39.0's mandatory CFG branch."""

    def __init__(self, dtype=torch.bfloat16):
        super().__init__()
        self.register_buffer("_dtype_anchor", torch.empty(0, dtype=dtype), persistent=False)

    @property
    def dtype(self):
        return self._dtype_anchor.dtype

    def forward(self, *, hidden_states, **kwargs):
        return (torch.zeros_like(hidden_states),)


transformer = Ideogram4Transformer2DModel.from_pretrained(
    repo_id,
    subfolder="transformer",
    torch_dtype=torch.bfloat16,
)
pipe = Ideogram4Pipeline.from_pretrained(
    components_repo_id,
    revision=components_revision,
    transformer=transformer,
    unconditional_transformer=None,
    torch_dtype=torch.bfloat16,
)
pipe.register_modules(unconditional_transformer=ZeroUnconditionalTransformer())
pipe.to("cuda")

prompt = json.dumps(
    {
        "high_level_description": (
            "A bold typographic poster centered on the exact words INSTANT BY FAL, "
            "printed in black and electric orange on warm white paper."
        ),
        "compositional_deconstruction": {
            "background": (
                "Warm white textured paper with even studio lighting and generous negative space."
            ),
            "elements": [
                {
                    "type": "text",
                    "text": "INSTANT BY FAL",
                    "desc": (
                        "Large uppercase geometric sans-serif lettering with crisp print edges, "
                        "precisely centered."
                    ),
                }
            ],
        },
    },
    ensure_ascii=False,
    separators=(",", ":"),
)

generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
    prompt,
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=1.0,
    guidance_schedule=None,
    mu=0.0,
    std=1.75,
    generator=generator,
).images[0]
image.save("ideogram4-instant.png")

The compatibility module has no parameters and its trivial zero output is multiplied by zero; no unconditional model is loaded and there is no effective runtime CFG.

Repository layout

.
├── README.md
├── LICENSE.md
├── NOTICE
├── assets/
│   └── ideogram-v4-by-fal.mp4
└── transformer/
    ├── config.json
    ├── diffusion_pytorch_model-00001-of-00004.safetensors
    ├── diffusion_pytorch_model-00002-of-00004.safetensors
    ├── diffusion_pytorch_model-00003-of-00004.safetensors
    ├── diffusion_pytorch_model-00004-of-00004.safetensors
    └── diffusion_pytorch_model.safetensors.index.json

Weights and provenance

This is the dense BF16 Instant checkpoint captured immediately before QAD. It is not a statically quantized FP4 or NVFP4 export. During conversion, fused QKV tensors were split into the standard Diffusers to_q, to_k, to_v, and to_out layout without changing their values.

The transformer was derived from ideogram-ai/ideogram-4-fp8. Shared inference components are loaded from ideogram-ai/ideogram-4-nf4-diffusers; neither transformer in that repository is loaded or used.

Ideogram 4 was created by Ideogram AI. This derivative checkpoint was developed and released by fal and is not an official Ideogram product or endorsed by Ideogram AI.

License

As a derivative of Ideogram 4, this model inherits the Ideogram 4 Non-Commercial Model Agreement. The complete inherited license is included in LICENSE.md and governs use and redistribution of this model.