diffusers/pr_12595/en/api/models/stable_audio_transformer.md

StableAudioDiTModel

A Transformer model for audio waveforms from Stable Audio Open.

StableAudioDiTModel[[diffusers.StableAudioDiTModel]]

class diffusers.StableAudioDiTModeldiffusers.StableAudioDiTModelhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/stable_audio_transformer.py#L185[{"name": "sample_size", "val": ": int = 1024"}, {"name": "in_channels", "val": ": int = 64"}, {"name": "num_layers", "val": ": int = 24"}, {"name": "attention_head_dim", "val": ": int = 64"}, {"name": "num_attention_heads", "val": ": int = 24"}, {"name": "num_key_value_attention_heads", "val": ": int = 12"}, {"name": "out_channels", "val": ": int = 64"}, {"name": "cross_attention_dim", "val": ": int = 768"}, {"name": "time_proj_dim", "val": ": int = 256"}, {"name": "global_states_input_dim", "val": ": int = 1536"}, {"name": "cross_attention_input_dim", "val": ": int = 768"}]- sample_size ( int, optional, defaults to 1024) -- The size of the input sample.

• in_channels (int, optional, defaults to 64) -- The number of channels in the input.
• num_layers (int, optional, defaults to 24) -- The number of layers of Transformer blocks to use.
• attention_head_dim (int, optional, defaults to 64) -- The number of channels in each head.
• num_attention_heads (int, optional, defaults to 24) -- The number of heads to use for the query states.
• num_key_value_attention_heads (int, optional, defaults to 12) -- The number of heads to use for the key and value states.
• out_channels (int, defaults to 64) -- Number of output channels.
• cross_attention_dim ( int, optional, defaults to 768) -- Dimension of the cross-attention projection.
• time_proj_dim ( int, optional, defaults to 256) -- Dimension of the timestep inner projection.
• global_states_input_dim ( int, optional, defaults to 1536) -- Input dimension of the global hidden states projection.
• cross_attention_input_dim ( int, optional, defaults to 768) -- Input dimension of the cross-attention projection0

The Diffusion Transformer model introduced in Stable Audio.

Reference: https://github.com/Stability-AI/stable-audio-tools

forwarddiffusers.StableAudioDiTModel.forwardhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/stable_audio_transformer.py#L344[{"name": "hidden_states", "val": ": FloatTensor"}, {"name": "timestep", "val": ": LongTensor = None"}, {"name": "encoder_hidden_states", "val": ": FloatTensor = None"}, {"name": "global_hidden_states", "val": ": FloatTensor = None"}, {"name": "rotary_embedding", "val": ": FloatTensor = None"}, {"name": "return_dict", "val": ": bool = True"}, {"name": "attention_mask", "val": ": typing.Optional[torch.LongTensor] = None"}, {"name": "encoder_attention_mask", "val": ": typing.Optional[torch.LongTensor] = None"}]- hidden_states (torch.FloatTensor of shape (batch size, in_channels, sequence_len)) -- Input hidden_states.

• timestep ( torch.LongTensor) -- Used to indicate denoising step.

• encoder_hidden_states (torch.FloatTensor of shape (batch size, encoder_sequence_len, cross_attention_input_dim)) -- Conditional embeddings (embeddings computed from the input conditions such as prompts) to use.

• global_hidden_states (torch.FloatTensor of shape (batch size, global_sequence_len, global_states_input_dim)) -- Global embeddings that will be prepended to the hidden states.

• rotary_embedding (torch.Tensor) -- The rotary embeddings to apply on query and key tensors during attention calculation.

• return_dict (bool, optional, defaults to True) -- Whether or not to return a ~models.transformer_2d.Transformer2DModelOutput instead of a plain tuple.

• attention_mask (torch.Tensor of shape (batch_size, sequence_len), optional) -- Mask to avoid performing attention on padding token indices, formed by concatenating the attention masks for the two text encoders together. Mask values selected in [0, 1]:

  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.

• encoder_attention_mask (torch.Tensor of shape (batch_size, sequence_len), optional) -- Mask to avoid performing attention on padding token cross-attention indices, formed by concatenating the attention masks for the two text encoders together. Mask values selected in [0, 1]:

  • 1 for tokens that are not masked,
  • 0 for tokens that are masked.0If return_dict is True, an ~models.transformer_2d.Transformer2DModelOutput is returned, otherwise a

tuple where the first element is the sample tensor.

The StableAudioDiTModel forward method.

set_attn_processordiffusers.StableAudioDiTModel.set_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/stable_audio_transformer.py#L303[{"name": "processor", "val": ": typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]]"}]- processor (dict of AttentionProcessor or only AttentionProcessor) -- The instantiated processor class or a dictionary of processor classes that will be set as the processor for all Attention layers.

If processor is a dict, the key needs to define the path to the corresponding cross attention processor. This is strongly recommended when setting trainable attention processors.0

Sets the attention processor to use to compute attention.

set_default_attn_processordiffusers.StableAudioDiTModel.set_default_attn_processorhttps://github.com/huggingface/diffusers/blob/vr_12595/src/diffusers/models/transformers/stable_audio_transformer.py#L338[]

Disables custom attention processors and sets the default attention implementation.