| --- |
| base_model: ACE-Step/Ace-Step1.5 |
| library_name: onnxruntime |
| tags: |
| - onnx |
| - webgpu |
| - music-generation |
| - text-to-music |
| - diffusion |
| - flow-matching |
| - ace-step |
| license: apache-2.0 |
| pipeline_tag: text-to-audio |
| --- |
| |
| # ACE-Step v1.5 — ONNX |
|
|
| ONNX export of [ACE-Step/Ace-Step1.5](https://huggingface.co/ACE-Step/Ace-Step1.5), a text-to-music generation model using flow matching with a Diffusion Transformer (DiT). |
|
|
| Exported for WebGPU inference via [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/). |
|
|
| ## Model Components |
|
|
| ACE-Step v1.5 consists of several components that work together: |
|
|
| | Component | Description | FP32 | INT4 | FP16 | |
| |---|---|---|---|---| |
| | **DiT decoder** | Main diffusion transformer (24 layers, 2048 hidden, 8-step turbo) | 6.3 GB | 2.1 GB | — | |
| | **LM** (1.7B) | Causal language model for lyric-conditioned generation | 7.4 GB | 5.1 GB | — | |
| | **Text encoder** (0.6B) | Qwen3-Embedding for text conditioning | 2.4 GB | 1.7 GB | — | |
| | **Lyric encoder** | 8-layer transformer for lyric embeddings | 1.6 GB | 216 MB | — | |
| | **Timbre encoder** | 4-layer transformer for reference audio timbre | 806 MB | 108 MB | — | |
| | **VAE decoder** | AutoencoderOobleck (latent → stereo 48kHz waveform) | 337 MB | — | 169 MB | |
| | **Text projector** | Linear projection (1024 → 2048) | 8 MB | — | 4 MB | |
| | **Embed tokens** | Embedding table lookup for lyrics | 621 MB | — | 311 MB | |
|
|
| ## Directory Structure |
|
|
| ``` |
| onnx/ # FP32 ONNX models (full precision, for validation) |
| onnx_q4/ # INT4 weight-only quantized (for WebGPU deployment) |
| onnx_fp16/ # FP16 models (for conv-heavy / small components) |
| ``` |
|
|
| ## Usage for WebGPU |
|
|
| For text-to-music generation without the LM, the minimum model set is: |
| - `onnx_q4/dit_decoder_q4.onnx` (2.1 GB) |
| - `onnx_q4/text_encoder_q4.onnx` (1.7 GB) |
| - `onnx_fp16/text_embed_tokens_fp16.onnx` (311 MB) |
| - `onnx_q4/lyric_encoder_q4.onnx` (216 MB) |
| - `onnx_fp16/vae_decoder_fp16.onnx` (169 MB) |
| - `onnx_q4/timbre_encoder_q4.onnx` (108 MB) |
| - `onnx_fp16/text_projector_fp16.onnx` (4 MB) |
|
|
| **Total: ~4.6 GB** — fits in 8 GB VRAM on desktop GPUs. |
|
|
| ## Inference Pipeline |
|
|
| 1. **Text encoding**: Tokenize caption → text_encoder → text_projector |
| 2. **Lyric encoding**: Tokenize lyrics → embed_tokens → lyric_encoder |
| 3. **Timbre encoding**: Reference audio latents → timbre_encoder |
| 4. **Condition packing**: Concatenate and pack text + lyric + timbre embeddings (JS logic) |
| 5. **Denoising loop** (8 steps): DiT decoder with Euler ODE scheduler |
| 6. **VAE decode**: Latents → stereo 48kHz waveform |
| |
| The flow-matching scheduler runs in JavaScript — only the DiT forward pass is in ONNX. |
| |
| ## Technical Details |
| |
| - **Latent space**: 64 channels, 25 Hz frame rate (1920x upsampling to 48kHz) |
| - **Denoising**: 8-step turbo schedule with flow matching (Euler ODE) |
| - **Attention**: Alternating full + sliding-window (128) bidirectional attention with GQA (16 query / 8 KV heads) |
| - **Quantization**: INT4 weight-only (MatMulNBits, block_size=128, symmetric) |
|
|
| ## Export Verification |
|
|
| All exports verified against PyTorch reference with max absolute differences: |
|
|
| | Component | Max Diff | |
| |---|---| |
| | VAE decoder | 9.2e-6 | |
| | Text encoder | 2.3e-4 | |
| | Embed tokens | 0.0 (exact) | |
| | DiT decoder | 2.2e-5 | |
| | LM | 3.2e-3 | |
| | Lyric encoder | 2.4e-5 | |
| | Timbre encoder | 1.7e-5 | |
| | Text projector | 3.6e-6 | |
|
|
| ## Attribution |
|
|
| This is an ONNX conversion of [ACE-Step v1.5](https://huggingface.co/ACE-Step/Ace-Step1.5) by the [ACE-Step team](https://github.com/ace-step). |
|
|
| - **Paper**: [ACE-Step: A Step Towards Music Generation Foundation Model](https://arxiv.org/abs/2506.00045) |
| - **Code**: [github.com/ace-step/ACE-Step-1.5](https://github.com/ace-step/ACE-Step-1.5) |
| - **License**: Apache 2.0 |
|
|
