UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion

       

---

UNISON is a unified latent flow-matching framework for audio and speech generation and editing. Using a single set of weights, it integrates text-to-audio, text-to-speech, zero-shot speaker cloning, mixed speech-and-sound scene generation, and audio/speech-in-scene editing — all in one model, one architecture, one forward pass.

---

Model variants in this repository

This repository hosts two checkpoint variants:

Directory	VAE	DiT depth	Channels	Config
unison_D20S0_O_40ch/	MMAudio 44 kHz	20 double + 0 single	40	D20S0_O_40ch.yaml
unison_D24S0_O_20ch/	MMAudio 16 kHz	24 double + 0 single	20	D24S0_O_20ch.yaml

Both variants share the same Qwen2.5-Omni-7B text encoder and the same inference pipeline.

---

Supported tasks

Task	Prompt format
Text-to-Audio (T2A)	[Audio] {caption}
Text-to-Speech (TTS)	[Speech] A {female/male} voice saying "{text}"
Mixed Speech + Sound	[Speech] A {gender} voice saying "{text}" [Audio] {background}
Zero-shot Speaker Cloning	[Speech with voice] {ref_text}, {target_text}
Audio Scene Editing (add / remove / replace / denoise)	[Edit] [Audio] {instruction}
Speech-in-Scene Editing (content / insert / delete)	[Edit] [Speech] {instruction}
Timed Temporal Composition	[Audio] From {t1}s to {t2}s, {event1}. From {t2}s to {t3}s, {event2}. ...

Task identity is encoded via a mask channel; source/reference audio is injected through VAE-encoded channel concatenation — no separate encoders or task-specific heads needed.

---

Architecture

All tasks share the same VAE encoder/decoder, MM-DiT backbone, and forward pass. Text conditioning uses layer-wise deep LLM fusion: hidden states from uniformly sampled layers of the frozen Qwen2.5-Omni-7B backbone are injected into corresponding MM-DiT double-stream blocks via learned linear projections.

---

Quick start

1. Clone repo and install dependencies

git clone https://github.com/lizhaoqing/UNISON
cd UNISON
pip install -r requirements.txt

flash-attn is optional but strongly recommended (automatic fallback to PyTorch SDPA):

pip install flash-attn --no-build-isolation

2. MMAudio VAE weights

Download from the MMAudio release and place at:

unison/models/mmaudio/data/ext_weights/
    v1-44.pth       # 44 kHz VAE  (for D20S0 / 44k variant)
    v1-16.pth       # 16 kHz VAE  (for D24S0 / 16k variant)
    best_netG.pt    # BigVGAN vocoder  (16 kHz VAE only)

3. Qwen2.5-Omni-7B

export QWEN_OMNI_MODEL_PATH=Qwen/Qwen2.5-Omni-7B
# or point to a local download:
export QWEN_OMNI_MODEL_PATH=/path/to/Qwen2.5-Omni-7B

4. Download checkpoints (this repo)

from huggingface_hub import snapshot_download
snapshot_download(repo_id="jac22/UNISON", local_dir="checkpoints")

This produces:

checkpoints/
    unison_D20S0_O_40ch/model.safetensors   # 44 kHz
    unison_D24S0_O_20ch/model.safetensors   # 16 kHz

5. Run inference

cd UNISON

# 44 kHz variant (D20S0)
bash scripts/infer.sh \
    --checkpoint_dir checkpoints/unison_D20S0_O_40ch \
    --model_config   unison/config/D20S0_O_40ch.yaml \
    --vae_config     unison/models/mmaudio/vae_config_44k.yaml \
    --task_mode      all

# 16 kHz variant (D24S0)
bash scripts/infer.sh \
    --checkpoint_dir checkpoints/unison_D24S0_O_20ch \
    --model_config   unison/config/D24S0_O_20ch.yaml \
    --vae_config     unison/models/mmaudio/vae_config_16k.yaml \
    --task_mode      all

Outputs are written to <checkpoint_dir>/infer_<N>steps/<ckpt_name>/.

Single-prompt example

python unison/pipelines/infer.py \
  --model_ckpt      checkpoints/unison_D20S0_O_40ch \
  --model_config    unison/config/D20S0_O_40ch.yaml \
  --vae_config      unison/models/mmaudio/vae_config_44k.yaml \
  --omni_model_path $QWEN_OMNI_MODEL_PATH \
  --task_mode       generation \
  --gen_prompt      "[Audio] Rain falling on a tin roof with distant thunder" \
  --gen_duration    10.0 \
  --output_dir      outputs/demo

---

Key inference parameters

Argument	Default	Description
--num_inference_steps	100	ODE solver steps (50 for fast, 100 for paper quality)
--guidance_scale	4.5	Classifier-free guidance scale
--seed	42	Random seed
--gen_duration	10.0	Output length in seconds (generation tasks)
--ref_duration	3.0	Reference clip length in seconds (zero-shot TTS)

---

Checkpoint format

Each checkpoint is a single model.safetensors file (unwrapped from EMA). The inference pipeline also accepts:

• A directory — auto-detects ema_model.pt → model.safetensors → pytorch_model.bin
• A direct file path to any of the three formats

EMA wrappers are unwrapped automatically at load time.

---

License

This project is released under the Apache 2.0 License with additional non-commercial use restrictions inherited from upstream dependencies:

• The backbone architecture derives from HunyuanVideo (Tencent), which prohibits commercial use without a separate license.
• Text/audio conditioning uses Qwen2.5-Omni (Alibaba Cloud), subject to its own license terms.

This model is intended for research and non-commercial use only.

---

Citation

@article{li2026unison,
  title   = {UNISON: A Unified Sound Generation and Editing Framework via Deep LLM Fusion},
  author  = {Li, Zhaoqing and Xu, Haoning and Su, Jingran and Liu, Yaofang and Rao, Zhefan and
             Wang, Huimeng and Deng, Jiajun and Wang, Tianzi and Jin, Zengrui and Liu, Rui and
             Che, Haoxuan and Liu, Xunying},
  journal = {arXiv preprint arXiv:2605.31530},
  year    = {2026}
}

---

Acknowledgements

We thank the authors of the following works for their excellent open-source contributions:

• HunyuanVideo — MM-DiT backbone architecture
• MMAudio — audio VAE and feature utilities
• Qwen2.5-Omni — text/audio LLM used for deep conditioning
• Ovi (Character.AI) — inspiring cross-modal fusion design for joint audio-video generation