Hugging Face's logo

Diffusers documentation

HunyuanVideo15Transformer3DModel

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

HunyuanVideo15Transformer3DModel

A Diffusion Transformer model for 3D video-like data used in HunyuanVideo1.5.

The model can be loaded with the following code snippet.

from diffusers import HunyuanVideo15Transformer3DModel

transformer = HunyuanVideo15Transformer3DModel.from_pretrained("hunyuanvideo-community/HunyuanVideo-1.5-Diffusers-480p_t2v" subfolder="transformer", torch_dtype=torch.bfloat16)

HunyuanVideo15Transformer3DModel

class diffusers.HunyuanVideo15Transformer3DModel

< >

( in_channels: int = 65 out_channels: int = 32 num_attention_heads: int = 16 attention_head_dim: int = 128 num_layers: int = 54 num_refiner_layers: int = 2 mlp_ratio: float = 4.0 patch_size: int = 1 patch_size_t: int = 1 qk_norm: str = 'rms_norm' text_embed_dim: int = 3584 text_embed_2_dim: int = 1472 image_embed_dim: int = 1152 rope_theta: float = 256.0 rope_axes_dim: tuple = (16, 56, 56) target_size: int = 640 task_type: str = 'i2v' use_meanflow: bool = False )

Parameters

  • in_channels (int, defaults to 16) — The number of channels in the input.
  • out_channels (int, defaults to 16) — The number of channels in the output.
  • num_attention_heads (int, defaults to 24) — The number of heads to use for multi-head attention.
  • attention_head_dim (int, defaults to 128) — The number of channels in each head.
  • num_layers (int, defaults to 20) — The number of layers of dual-stream blocks to use.
  • num_refiner_layers (int, defaults to 2) — The number of layers of refiner blocks to use.
  • mlp_ratio (float, defaults to 4.0) — The ratio of the hidden layer size to the input size in the feedforward network.
  • patch_size (int, defaults to 2) — The size of the spatial patches to use in the patch embedding layer.
  • patch_size_t (int, defaults to 1) — The size of the tmeporal patches to use in the patch embedding layer.
  • qk_norm (str, defaults to rms_norm) — The normalization to use for the query and key projections in the attention layers.
  • guidance_embeds (bool, defaults to True) — Whether to use guidance embeddings in the model.
  • text_embed_dim (int, defaults to 4096) — Input dimension of text embeddings from the text encoder.
  • pooled_projection_dim (int, defaults to 768) — The dimension of the pooled projection of the text embeddings.
  • rope_theta (float, defaults to 256.0) — The value of theta to use in the RoPE layer.
  • rope_axes_dim (tuple[int], defaults to (16, 56, 56)) — The dimensions of the axes to use in the RoPE layer.

A Transformer model for video-like data used in HunyuanVideo1.5.

forward

< >

( hidden_states: Tensor timestep: LongTensor encoder_hidden_states: Tensor encoder_attention_mask: Tensor timestep_r: torch.LongTensor | None = None encoder_hidden_states_2: torch.Tensor | None = None encoder_attention_mask_2: torch.Tensor | None = None image_embeds: torch.Tensor | None = None attention_kwargs: dict[str, typing.Any] | None = None return_dict: bool = True )

Parameters

  • hidden_states (torch.Tensor of shape (batch_size, num_channels, num_frames, height, width)) — Input hidden_states.
  • timestep (torch.LongTensor) — Used to indicate denoising step.
  • encoder_hidden_states (torch.Tensor of shape (batch_size, sequence_len, embed_dims)) — Conditional embeddings (embeddings computed from the input conditions such as prompts) to use.
  • encoder_attention_mask (torch.Tensor) — Mask applied to encoder_hidden_states during attention.
  • timestep_r (torch.LongTensor, optional) — Refiner timestep conditioning.
  • encoder_hidden_states_2 (torch.Tensor, optional) — Additional conditional embeddings computed from a second text encoder (ByT5).
  • encoder_attention_mask_2 (torch.Tensor, optional) — Mask applied to encoder_hidden_states_2 during attention.
  • image_embeds (torch.Tensor, optional) — Image embeddings for image-conditioned generation.
  • attention_kwargs (dict, optional) — A kwargs dictionary that if specified is passed along to the AttentionProcessor as defined under self.processor in diffusers.models.attention_processor.
  • return_dict (bool, optional, defaults to True) — Whether or not to return a ~models.transformer_2d.Transformer2DModelOutput instead of a plain tuple.

The HunyuanVideo15Transformer3DModel forward method.

Transformer2DModelOutput

class diffusers.models.modeling_outputs.Transformer2DModelOutput

< >

( sample: torch.Tensor )

Parameters

  • sample (torch.Tensor of shape (batch_size, num_channels, height, width) or (batch size, num_vector_embeds - 1, num_latent_pixels) if Transformer2DModel is discrete) — The hidden states output conditioned on the encoder_hidden_states input. If discrete, returns probability distributions for the unnoised latent pixels.

The output of Transformer2DModel.

Update on GitHub

←HunyuanImageTransformer2DModel HunyuanVideoTransformer3DModel→