diffusers/pr_13390/en/api/pipelines/longcat_audio_dit.md

LongCat-AudioDiT

LongCat-AudioDiT is a text-to-audio diffusion model from Meituan LongCat. The diffusers integration exposes a standard DiffusionPipeline interface for text-conditioned audio generation.

This pipeline supports loading the original flat LongCat checkpoint layout from either a local directory or a Hugging Face Hub repository containing:

• config.json
• model.safetensors

The loader builds the text encoder, transformer, and VAE from config.json, restores component weights from model.safetensors, and ties the shared UMT5 embedding when needed.

This pipeline was adapted from the LongCat-AudioDiT reference implementation: https://github.com/meituan-longcat/LongCat-AudioDiT

Usage

import soundfile as sf
import torch
from diffusers import LongCatAudioDiTPipeline

pipeline = LongCatAudioDiTPipeline.from_pretrained(
    "meituan-longcat/LongCat-AudioDiT-1B",
    torch_dtype=torch.float16,
)
pipeline = pipeline.to("cuda")

audio = pipeline(
    prompt="A calm ocean wave ambience with soft wind in the background.",
    audio_end_in_s=5.0,
    num_inference_steps=16,
    guidance_scale=4.0,
    output_type="pt",
).audios

output = audio[0, 0].float().cpu().numpy()
sf.write("longcat.wav", output, pipeline.sample_rate)

Tips

• audio_end_in_s is the most direct way to control output duration.
• output_type="pt" returns a PyTorch tensor shaped (batch, channels, samples).

LongCatAudioDiTPipeline[[diffusers.LongCatAudioDiTPipeline]]

diffusers.LongCatAudioDiTPipeline[[diffusers.LongCatAudioDiTPipeline]]

Source

__call__diffusers.LongCatAudioDiTPipeline.__call__https://github.com/huggingface/diffusers/blob/vr_13390/src/diffusers/pipelines/longcat_audio_dit/pipeline_longcat_audio_dit.py#L437[{"name": "prompt", "val": ": str | list[str]"}, {"name": "negative_prompt", "val": ": str | list[str] | None = None"}, {"name": "audio_duration_s", "val": ": float | None = None"}, {"name": "latents", "val": ": torch.Tensor | None = None"}, {"name": "num_inference_steps", "val": ": int = 16"}, {"name": "guidance_scale", "val": ": float = 4.0"}, {"name": "generator", "val": ": torch._C.Generator | list[torch._C.Generator] | None = None"}, {"name": "output_type", "val": ": str = 'np'"}, {"name": "return_dict", "val": ": bool = True"}, {"name": "callback_on_step_end", "val": ": typing.Optional[typing.Callable[[int, int], NoneType]] = None"}, {"name": "callback_on_step_end_tensor_inputs", "val": ": list = ['latents']"}]- prompt (str or list[str]) -- Prompt or prompts that guide audio generation.

• negative_prompt (str or list[str], optional) -- Negative prompt(s) for classifier-free guidance.
• audio_duration_s (float, optional) -- Target audio duration in seconds. Ignored when latents is provided.
• latents (torch.Tensor, optional) -- Pre-generated noisy latents of shape (batch_size, duration, latent_dim).
• num_inference_steps (int, defaults to 16) -- Number of denoising steps.
• guidance_scale (float, defaults to 4.0) -- Guidance scale for classifier-free guidance.
• generator (torch.Generator or list[torch.Generator], optional) -- Random generator(s).
• output_type (str, defaults to "np") -- Output format: "np", "pt", or "latent".
• return_dict (bool, defaults to True) -- Whether to return AudioPipelineOutput.
• callback_on_step_end (Callable, optional) -- A function called at the end of each denoising step with the pipeline, step index, timestep, and tensor inputs specified by callback_on_step_end_tensor_inputs.
• callback_on_step_end_tensor_inputs (list, defaults to ["latents"]) -- Tensor inputs passed to callback_on_step_end.0

Function invoked when calling the pipeline for generation.

Parameters:

prompt (str or list[str]) : Prompt or prompts that guide audio generation.

negative_prompt (str or list[str], optional) : Negative prompt(s) for classifier-free guidance.

audio_duration_s (float, optional) : Target audio duration in seconds. Ignored when latents is provided.

latents (torch.Tensor, optional) : Pre-generated noisy latents of shape (batch_size, duration, latent_dim).

num_inference_steps (int, defaults to 16) : Number of denoising steps.

guidance_scale (float, defaults to 4.0) : Guidance scale for classifier-free guidance.

generator (torch.Generator or list[torch.Generator], optional) : Random generator(s).

output_type (str, defaults to "np") : Output format: "np", "pt", or "latent".

return_dict (bool, defaults to True) : Whether to return AudioPipelineOutput.

callback_on_step_end (Callable, optional) : A function called at the end of each denoising step with the pipeline, step index, timestep, and tensor inputs specified by callback_on_step_end_tensor_inputs.

callback_on_step_end_tensor_inputs (list, defaults to ["latents"]) : Tensor inputs passed to callback_on_step_end.

from_pretrained[[diffusers.LongCatAudioDiTPipeline.from_pretrained]]

Source