diffusers/pr_12229/en/api/models/hunyuan_transformer2d.md

HunyuanDiT2DModel

A Diffusion Transformer model for 2D data from Hunyuan-DiT.

HunyuanDiT2DModel[[diffusers.HunyuanDiT2DModel]]

class diffusers.HunyuanDiT2DModeldiffusers.HunyuanDiT2DModelhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L202[{"name": "num_attention_heads", "val": ": int = 16"}, {"name": "attention_head_dim", "val": ": int = 88"}, {"name": "in_channels", "val": ": typing.Optional[int] = None"}, {"name": "patch_size", "val": ": typing.Optional[int] = None"}, {"name": "activation_fn", "val": ": str = 'gelu-approximate'"}, {"name": "sample_size", "val": " = 32"}, {"name": "hidden_size", "val": " = 1152"}, {"name": "num_layers", "val": ": int = 28"}, {"name": "mlp_ratio", "val": ": float = 4.0"}, {"name": "learn_sigma", "val": ": bool = True"}, {"name": "cross_attention_dim", "val": ": int = 1024"}, {"name": "norm_type", "val": ": str = 'layer_norm'"}, {"name": "cross_attention_dim_t5", "val": ": int = 2048"}, {"name": "pooled_projection_dim", "val": ": int = 1024"}, {"name": "text_len", "val": ": int = 77"}, {"name": "text_len_t5", "val": ": int = 256"}, {"name": "use_style_cond_and_image_meta_size", "val": ": bool = True"}]- num_attention_heads (int, optional, defaults to 16) -- The number of heads to use for multi-head attention.

• attention_head_dim (int, optional, defaults to 88) -- The number of channels in each head.
• in_channels (int, optional) -- The number of channels in the input and output (specify if the input is continuous).
• patch_size (int, optional) -- The size of the patch to use for the input.
• activation_fn (str, optional, defaults to "geglu") -- Activation function to use in feed-forward.
• sample_size (int, optional) -- The width of the latent images. This is fixed during training since it is used to learn a number of position embeddings.
• dropout (float, optional, defaults to 0.0) -- The dropout probability to use.
• cross_attention_dim (int, optional) -- The number of dimension in the clip text embedding.
• hidden_size (int, optional) -- The size of hidden layer in the conditioning embedding layers.
• num_layers (int, optional, defaults to 1) -- The number of layers of Transformer blocks to use.
• mlp_ratio (float, optional, defaults to 4.0) -- The ratio of the hidden layer size to the input size.
• learn_sigma (bool, optional, defaults to True) -- Whether to predict variance.
• cross_attention_dim_t5 (int, optional) -- The number dimensions in t5 text embedding.
• pooled_projection_dim (int, optional) -- The size of the pooled projection.
• text_len (int, optional) -- The length of the clip text embedding.
• text_len_t5 (int, optional) -- The length of the T5 text embedding.
• use_style_cond_and_image_meta_size (bool, optional) -- Whether or not to use style condition and image meta size. True for version <=1.1, False for version >= 1.20

HunYuanDiT: Diffusion model with a Transformer backbone.

Inherit ModelMixin and ConfigMixin to be compatible with the sampler StableDiffusionPipeline of diffusers.

enable_forward_chunkingdiffusers.HunyuanDiT2DModel.enable_forward_chunkinghttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L531[{"name": "chunk_size", "val": ": typing.Optional[int] = None"}, {"name": "dim", "val": ": int = 0"}]- chunk_size (int, optional) -- The chunk size of the feed-forward layers. If not specified, will run feed-forward layer individually over each tensor of dim=dim.

• dim (int, optional, defaults to 0) -- The dimension over which the feed-forward computation should be chunked. Choose between dim=0 (batch) or dim=1 (sequence length).0

Sets the attention processor to use feed forward chunking.

forwarddiffusers.HunyuanDiT2DModel.forwardhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L418[{"name": "hidden_states", "val": ""}, {"name": "timestep", "val": ""}, {"name": "encoder_hidden_states", "val": " = None"}, {"name": "text_embedding_mask", "val": " = None"}, {"name": "encoder_hidden_states_t5", "val": " = None"}, {"name": "text_embedding_mask_t5", "val": " = None"}, {"name": "image_meta_size", "val": " = None"}, {"name": "style", "val": " = None"}, {"name": "image_rotary_emb", "val": " = None"}, {"name": "controlnet_block_samples", "val": " = None"}, {"name": "return_dict", "val": " = True"}]- hidden_states (torch.Tensor of shape (batch size, dim, height, width)) -- The input tensor.

• timestep ( torch.LongTensor, optional) -- Used to indicate denoising step.
• encoder_hidden_states ( torch.Tensor of shape (batch size, sequence len, embed dims), optional) -- Conditional embeddings for cross attention layer. This is the output of BertModel.
• text_embedding_mask -- torch.Tensor An attention mask of shape (batch, key_tokens) is applied to encoder_hidden_states. This is the output of BertModel.
• encoder_hidden_states_t5 ( torch.Tensor of shape (batch size, sequence len, embed dims), optional) -- Conditional embeddings for cross attention layer. This is the output of T5 Text Encoder.
• text_embedding_mask_t5 -- torch.Tensor An attention mask of shape (batch, key_tokens) is applied to encoder_hidden_states. This is the output of T5 Text Encoder.
• image_meta_size (torch.Tensor) -- Conditional embedding indicate the image sizes
• style -- torch.Tensor: Conditional embedding indicate the style
• image_rotary_emb (torch.Tensor) -- The image rotary embeddings to apply on query and key tensors during attention calculation.
• return_dict -- bool Whether to return a dictionary.0

The HunyuanDiT2DModel forward method.

fuse_qkv_projectionsdiffusers.HunyuanDiT2DModel.fuse_qkv_projectionshttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L321[]

Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused.

> This API is 🧪 experimental.

set_attn_processordiffusers.HunyuanDiT2DModel.set_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L378[{"name": "processor", "val": ": typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]]"}]- processor (dict of AttentionProcessor or only AttentionProcessor) -- The instantiated processor class or a dictionary of processor classes that will be set as the processor for all Attention layers.

If processor is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.0

Sets the attention processor to use to compute attention.

set_default_attn_processordiffusers.HunyuanDiT2DModel.set_default_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L412[]

Disables custom attention processors and sets the default attention implementation.

unfuse_qkv_projectionsdiffusers.HunyuanDiT2DModel.unfuse_qkv_projectionshttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/hunyuan_transformer_2d.py#L343[] Disables the fused QKV projection if enabled.

> This API is 🧪 experimental.