---
base_model: ACE-Step/Ace-Step1.5
library_name: onnxruntime
tags:
  - onnx
  - webgpu
  - music-generation
  - text-to-music
  - diffusion
  - flow-matching
  - ace-step
license: apache-2.0
pipeline_tag: text-to-audio
---

# ACE-Step v1.5 — ONNX

ONNX export of [ACE-Step/Ace-Step1.5](https://huggingface.co/ACE-Step/Ace-Step1.5), a text-to-music generation model using flow matching with a Diffusion Transformer (DiT).

Exported for WebGPU inference via [ONNX Runtime Web](https://onnxruntime.ai/docs/tutorials/web/).

## Model Components

ACE-Step v1.5 consists of several components that work together:

| Component | Description | FP32 | INT4 | FP16 |
|---|---|---|---|---|
| **DiT decoder** | Main diffusion transformer (24 layers, 2048 hidden, 8-step turbo) | 6.3 GB | 2.1 GB | — |
| **LM** (1.7B) | Causal language model for lyric-conditioned generation | 7.4 GB | 5.1 GB | — |
| **Text encoder** (0.6B) | Qwen3-Embedding for text conditioning | 2.4 GB | 1.7 GB | — |
| **Lyric encoder** | 8-layer transformer for lyric embeddings | 1.6 GB | 216 MB | — |
| **Timbre encoder** | 4-layer transformer for reference audio timbre | 806 MB | 108 MB | — |
| **VAE decoder** | AutoencoderOobleck (latent → stereo 48kHz waveform) | 337 MB | — | 169 MB |
| **Text projector** | Linear projection (1024 → 2048) | 8 MB | — | 4 MB |
| **Embed tokens** | Embedding table lookup for lyrics | 621 MB | — | 311 MB |

## Directory Structure

```
onnx/          # FP32 ONNX models (full precision, for validation)
onnx_q4/       # INT4 weight-only quantized (for WebGPU deployment)
onnx_fp16/     # FP16 models (for conv-heavy / small components)
```

## Usage for WebGPU

For text-to-music generation without the LM, the minimum model set is:
- `onnx_q4/dit_decoder_q4.onnx` (2.1 GB)
- `onnx_q4/text_encoder_q4.onnx` (1.7 GB)
- `onnx_fp16/text_embed_tokens_fp16.onnx` (311 MB)
- `onnx_q4/lyric_encoder_q4.onnx` (216 MB)
- `onnx_fp16/vae_decoder_fp16.onnx` (169 MB)
- `onnx_q4/timbre_encoder_q4.onnx` (108 MB)
- `onnx_fp16/text_projector_fp16.onnx` (4 MB)

**Total: ~4.6 GB** — fits in 8 GB VRAM on desktop GPUs.

## Inference Pipeline

1. **Text encoding**: Tokenize caption → text_encoder → text_projector
2. **Lyric encoding**: Tokenize lyrics → embed_tokens → lyric_encoder
3. **Timbre encoding**: Reference audio latents → timbre_encoder
4. **Condition packing**: Concatenate and pack text + lyric + timbre embeddings (JS logic)
5. **Denoising loop** (8 steps): DiT decoder with Euler ODE scheduler
6. **VAE decode**: Latents → stereo 48kHz waveform

The flow-matching scheduler runs in JavaScript — only the DiT forward pass is in ONNX.

## Technical Details

- **Latent space**: 64 channels, 25 Hz frame rate (1920x upsampling to 48kHz)
- **Denoising**: 8-step turbo schedule with flow matching (Euler ODE)
- **Attention**: Alternating full + sliding-window (128) bidirectional attention with GQA (16 query / 8 KV heads)
- **Quantization**: INT4 weight-only (MatMulNBits, block_size=128, symmetric)

## Export Verification

All exports verified against PyTorch reference with max absolute differences:

| Component | Max Diff |
|---|---|
| VAE decoder | 9.2e-6 |
| Text encoder | 2.3e-4 |
| Embed tokens | 0.0 (exact) |
| DiT decoder | 2.2e-5 |
| LM | 3.2e-3 |
| Lyric encoder | 2.4e-5 |
| Timbre encoder | 1.7e-5 |
| Text projector | 3.6e-6 |

## Attribution

This is an ONNX conversion of [ACE-Step v1.5](https://huggingface.co/ACE-Step/Ace-Step1.5) by the [ACE-Step team](https://github.com/ace-step).

- **Paper**: [ACE-Step: A Step Towards Music Generation Foundation Model](https://arxiv.org/abs/2506.00045)
- **Code**: [github.com/ace-step/ACE-Step-1.5](https://github.com/ace-step/ACE-Step-1.5)
- **License**: Apache 2.0