diffusers/pr_12229/en/api/models/pixart_transformer2d.md

PixArtTransformer2DModel

A Transformer model for image-like data from PixArt-Alpha and PixArt-Sigma.

PixArtTransformer2DModel[[diffusers.PixArtTransformer2DModel]]

class diffusers.PixArtTransformer2DModeldiffusers.PixArtTransformer2DModelhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L364[{"name": "num_attention_heads", "val": ": int = 16"}, {"name": "attention_head_dim", "val": ": int = 72"}, {"name": "in_channels", "val": ": int = 4"}, {"name": "out_channels", "val": ": typing.Optional[int] = 8"}, {"name": "num_layers", "val": ": int = 28"}, {"name": "dropout", "val": ": float = 0.0"}, {"name": "norm_num_groups", "val": ": int = 32"}, {"name": "cross_attention_dim", "val": ": typing.Optional[int] = 1152"}, {"name": "attention_bias", "val": ": bool = True"}, {"name": "sample_size", "val": ": int = 128"}, {"name": "patch_size", "val": ": int = 2"}, {"name": "activation_fn", "val": ": str = 'gelu-approximate'"}, {"name": "num_embeds_ada_norm", "val": ": typing.Optional[int] = 1000"}, {"name": "upcast_attention", "val": ": bool = False"}, {"name": "norm_type", "val": ": str = 'ada_norm_single'"}, {"name": "norm_elementwise_affine", "val": ": bool = False"}, {"name": "norm_eps", "val": ": float = 1e-06"}, {"name": "interpolation_scale", "val": ": typing.Optional[int] = None"}, {"name": "use_additional_conditions", "val": ": typing.Optional[bool] = None"}, {"name": "caption_channels", "val": ": typing.Optional[int] = None"}, {"name": "attention_type", "val": ": typing.Optional[str] = 'default'"}]- num_attention_heads (int, optional, defaults to 16) -- The number of heads to use for multi-head attention.

• attention_head_dim (int, optional, defaults to 72) -- The number of channels in each head.
• in_channels (int, defaults to 4) -- The number of channels in the input.
• out_channels (int, optional) -- The number of channels in the output. Specify this parameter if the output channel number differs from the input.
• num_layers (int, optional, defaults to 28) -- The number of layers of Transformer blocks to use.
• dropout (float, optional, defaults to 0.0) -- The dropout probability to use within the Transformer blocks.
• norm_num_groups (int, optional, defaults to 32) -- Number of groups for group normalization within Transformer blocks.
• cross_attention_dim (int, optional) -- The dimensionality for cross-attention layers, typically matching the encoder's hidden dimension.
• attention_bias (bool, optional, defaults to True) -- Configure if the Transformer blocks' attention should contain a bias parameter.
• sample_size (int, defaults to 128) -- The width of the latent images. This parameter is fixed during training.
• patch_size (int, defaults to 2) -- Size of the patches the model processes, relevant for architectures working on non-sequential data.
• activation_fn (str, optional, defaults to "gelu-approximate") -- Activation function to use in feed-forward networks within Transformer blocks.
• num_embeds_ada_norm (int, optional, defaults to 1000) -- Number of embeddings for AdaLayerNorm, fixed during training and affects the maximum denoising steps during inference.
• upcast_attention (bool, optional, defaults to False) -- If true, upcasts the attention mechanism dimensions for potentially improved performance.
• norm_type (str, optional, defaults to "ada_norm_zero") -- Specifies the type of normalization used, can be 'ada_norm_zero'.
• norm_elementwise_affine (bool, optional, defaults to False) -- If true, enables element-wise affine parameters in the normalization layers.
• norm_eps (float, optional, defaults to 1e-6) -- A small constant added to the denominator in normalization layers to prevent division by zero.
• interpolation_scale (int, optional) -- Scale factor to use during interpolating the position embeddings.
• use_additional_conditions (bool, optional) -- If we're using additional conditions as inputs.
• attention_type (str, optional, defaults to "default") -- Kind of attention mechanism to be used.
• caption_channels (int, optional, defaults to None) -- Number of channels to use for projecting the caption embeddings.
• use_linear_projection (bool, optional, defaults to False) -- Deprecated argument. Will be removed in a future version.
• num_vector_embeds (bool, optional, defaults to False) -- Deprecated argument. Will be removed in a future version.0

A 2D Transformer model as introduced in PixArt family of models (https://huggingface.co/papers/2310.00426, https://huggingface.co/papers/2403.04692).

forwarddiffusers.PixArtTransformer2DModel.forwardhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L619[{"name": "hidden_states", "val": ": Tensor"}, {"name": "encoder_hidden_states", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "timestep", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "added_cond_kwargs", "val": ": typing.Dict[str, torch.Tensor] = None"}, {"name": "cross_attention_kwargs", "val": ": typing.Dict[str, typing.Any] = None"}, {"name": "attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "encoder_attention_mask", "val": ": typing.Optional[torch.Tensor] = None"}, {"name": "return_dict", "val": ": bool = True"}]- hidden_states (torch.FloatTensor of shape (batch size, channel, height, width)) -- Input hidden_states.

• encoder_hidden_states (torch.FloatTensor of shape (batch size, sequence len, embed dims), optional) -- Conditional embeddings for cross attention layer. If not given, cross-attention defaults to self-attention.

• timestep (torch.LongTensor, optional) -- Used to indicate denoising step. Optional timestep to be applied as an embedding in AdaLayerNorm.

• added_cond_kwargs -- (Dict[str, Any], optional): Additional conditions to be used as inputs.

• cross_attention_kwargs ( Dict[str, Any], optional) -- A kwargs dictionary that if specified is passed along to the AttentionProcessor as defined under self.processor in diffusers.models.attention_processor.

• attention_mask ( torch.Tensor, optional) -- An attention mask of shape (batch, key_tokens) is applied to encoder_hidden_states. If 1 the mask is kept, otherwise if 0 it is discarded. Mask will be converted into a bias, which adds large negative values to the attention scores corresponding to "discard" tokens.

• encoder_attention_mask ( torch.Tensor, optional) -- Cross-attention mask applied to encoder_hidden_states. Two formats supported:

  • Mask (batch, sequence_length) True = keep, False = discard.
  • Bias (batch, 1, sequence_length) 0 = keep, -10000 = discard.

  If ndim == 2: will be interpreted as a mask, then converted into a bias consistent with the format above. This bias will be added to the cross-attention scores.

• return_dict (bool, optional, defaults to True) -- Whether or not to return a UNet2DConditionOutput instead of a plain tuple.0If return_dict is True, an ~models.transformer_2d.Transformer2DModelOutput is returned, otherwise a tuple where the first element is the sample tensor.

The PixArtTransformer2DModel forward method.

fuse_qkv_projectionsdiffusers.PixArtTransformer2DModel.fuse_qkv_projectionshttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L588[]

Enables fused QKV projections. For self-attention modules, all projection matrices (i.e., query, key, value) are fused. For cross-attention modules, key and value projection matrices are fused.

> This API is 🧪 experimental.

set_attn_processordiffusers.PixArtTransformer2DModel.set_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L545[{"name": "processor", "val": ": typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]]"}]- processor (dict of AttentionProcessor or only AttentionProcessor) -- The instantiated processor class or a dictionary of processor classes that will be set as the processor for all Attention layers.

If processor is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.0

Sets the attention processor to use to compute attention.

set_default_attn_processordiffusers.PixArtTransformer2DModel.set_default_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L579[]

Disables custom attention processors and sets the default attention implementation.

Safe to just use AttnProcessor() as PixArt doesn't have any exotic attention processors in default model.

unfuse_qkv_projectionsdiffusers.PixArtTransformer2DModel.unfuse_qkv_projectionshttps://github.com/huggingface/diffusers/blob/vr_12229/src/diffusers/models/transformers/pixart_transformer_2d.py#L610[] Disables the fused QKV projection if enabled.

> This API is 🧪 experimental.