Hugging Face's logo

moonbox-dev
/
MOSS-TTS-v1.5-MLX

Text-to-Speech
Safetensors
MLX
mlx-audio
moonbox
moss-tts
openmoss
voice-cloning
qwen3
Model card Files
xet
 Community

MOSS-TTS v1.5 MLX

MLX builds of MOSS-TTS v1.5, made by Moonbox for Apple Silicon Macs. Each precision is a folder; keep a folder's files together. Only the parts listed below are converted: the rest of the pipeline comes from the original repository unless the section below says otherwise.

Files

The last column is how far each build is from the original PyTorch model on the same inputs (see the section below for what was measured).

Precision File Size sha256 Checked against the original
bf16 bf16/added_tokens.json 0.7 kB 9dfb40d3a3f8 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/chat_template.jinja 0.4 kB 894e3ace884e not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/config.json 2.7 kB 32dc731f256e not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/merges.txt 1.7 MB 8831e4f1a044 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/model-00001-of-00004.safetensors 5.32 GB ad91aa5d7bc9 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/model-00002-of-00004.safetensors 5.30 GB 0dac139d80f7 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/model-00003-of-00004.safetensors 4.82 GB bbff7aa8ad80 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/model-00004-of-00004.safetensors 1.54 GB 2e1fe38aed3b not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/model.safetensors.index.json 42.2 kB 378fd1b3e608 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/processor_config.json 0.1 kB 6821d4805f10 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/special_tokens_map.json 0.6 kB 55cbd0665670 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/tokenizer.json 11.4 MB cb3c8fa82993 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/tokenizer_config.json 5.5 kB 87b1f37a4a73 not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
bf16 bf16/vocab.json 2.8 MB ca10d7e9fb3e not checkable on 24 GB: weights bit-identical to the original's bf16; 1 English clip: WER 0.0 %, speaker sim 0.985, DNSMOS 3.25 (original 0.0 %, 0.976, 3.27)
fp16 fp16/added_tokens.json 0.7 kB 9dfb40d3a3f8 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/chat_template.jinja 0.4 kB 894e3ace884e not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/config.json 2.7 kB 32dc731f256e not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/merges.txt 1.7 MB 8831e4f1a044 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/model-00001-of-00004.safetensors 5.32 GB 497cad154ce7 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/model-00002-of-00004.safetensors 5.30 GB 2d6ae14a1152 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/model-00003-of-00004.safetensors 4.82 GB fa025e974161 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/model-00004-of-00004.safetensors 1.54 GB 292e3b660c52 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/model.safetensors.index.json 42.2 kB 378fd1b3e608 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/processor_config.json 0.1 kB 6821d4805f10 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/special_tokens_map.json 0.6 kB 55cbd0665670 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/tokenizer.json 11.4 MB cb3c8fa82993 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/tokenizer_config.json 5.5 kB 87b1f37a4a73 not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
fp16 fp16/vocab.json 2.8 MB ca10d7e9fb3e not checkable on 24 GB: weights within 3e-8 of the original's (bf16 -> fp16 cast); fp16 compute not run
8bit 8bit/added_tokens.json 0.7 kB 9dfb40d3a3f8 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/chat_template.jinja 0.4 kB 894e3ace884e WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/config.json 2.9 kB 970f7f37fde4 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/merges.txt 1.7 MB 8831e4f1a044 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/model-00001-of-00002.safetensors 5.34 GB ecd2f5cbd205 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/model-00002-of-00002.safetensors 3.81 GB d123db90708d WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/model.safetensors.index.json 94.2 kB f56f182aafb9 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/processor_config.json 0.1 kB 6821d4805f10 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/special_tokens_map.json 0.6 kB 55cbd0665670 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/tokenizer.json 11.4 MB cb3c8fa82993 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/tokenizer_config.json 5.5 kB 87b1f37a4a73 WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
8bit 8bit/vocab.json 2.8 MB ca10d7e9fb3e WER 0.0 % / CER 0.0 %, speaker sim 0.970 / 0.955, DNSMOS 3.11 / 2.72 (original bf16: 0.0 % / 0.0 %, 0.968 / 0.950, 2.94 / 2.36)
6bit 6bit/added_tokens.json 0.7 kB 9dfb40d3a3f8 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/chat_template.jinja 0.4 kB 894e3ace884e WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/config.json 2.9 kB bb3781b20691 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/merges.txt 1.7 MB 8831e4f1a044 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/model-00001-of-00002.safetensors 5.34 GB 03772900b21c WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/model-00002-of-00002.safetensors 1.72 GB f3627c576919 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/model.safetensors.index.json 94.2 kB d6da35422aa9 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/processor_config.json 0.1 kB 6821d4805f10 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/special_tokens_map.json 0.6 kB 55cbd0665670 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/tokenizer.json 11.4 MB cb3c8fa82993 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/tokenizer_config.json 5.5 kB 87b1f37a4a73 WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
6bit 6bit/vocab.json 2.8 MB ca10d7e9fb3e WER 0.0 % / CER 0.0 %, speaker sim 0.961 / 0.942, DNSMOS 3.09 / 2.52
4bit 4bit/added_tokens.json 0.7 kB 9dfb40d3a3f8 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/chat_template.jinja 0.4 kB 894e3ace884e below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/config.json 2.9 kB 83f1472b5354 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/merges.txt 1.7 MB 8831e4f1a044 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/model.safetensors 4.97 GB b3c2f49dd7d4 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/model.safetensors.index.json 78.7 kB 416cd7844336 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/processor_config.json 0.1 kB 6821d4805f10 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/special_tokens_map.json 0.6 kB 55cbd0665670 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/tokenizer.json 11.4 MB cb3c8fa82993 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/tokenizer_config.json 5.5 kB 87b1f37a4a73 below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79
4bit 4bit/vocab.json 2.8 MB ca10d7e9fb3e below threshold (experimental): WER 3.6 % / CER 1.3 %, speaker sim 0.967 / 0.943, DNSMOS 3.14 / 2.79

How to run

MOSS-TTS v1.5 (OpenMOSS, 8B MossTTSDelay on a Qwen3 backbone: zero-shot voice cloning from reference audio alone, many languages) for mlx-audio. Each folder is a complete mlx-audio model except the audio tokenizer: mlx-audio fetches OpenMOSS-Team/MOSS-Audio-Tokenizer (1.6B, 7 GB float32) at first use, or takes a local copy via audio_tokenizer_source. The original's remote-code .py files are left out (mlx-audio has its own implementation). Tested with mlx-audio==0.6.5:

from mlx_audio.tts.utils import load_model
import numpy as np, soundfile as sf

model = load_model("8bit")              # a downloaded folder
res = list(model.generate(text="Hello from Moonbox.", ref_audio="voice.wav", language="English"))
sf.write("out.wav", np.concatenate([np.array(r.audio).reshape(-1) for r in res]), res[0].sample_rate)

8bit/, 6bit/ and 4bit/ quantize the transformer layers (MLX affine, group size 64); embeddings and audio heads stay bf16. License: Apache-2.0.

How they were checked. Voice cloning of 6 English and 4 Chinese sentences, one seed each (the 8B model plus its audio tokenizer fill a 24 GB Mac), with the original (its own transformers code at the pinned revision, PyTorch bfloat16 on MPS, the same audio tokenizer) and every MLX build, the model's own sampling defaults. Judged by Whisper large-v3-turbo WER / CER, WavLM speaker similarity to the prompt, DNSMOS. Pass: error rate at most 2 points above the original's, similarity at most 0.03 below, DNSMOS at most 0.2 below.

Precision Size WER en / CER zh Speaker similarity en / zh DNSMOS en / zh Real-time factor, M5 24 GB Peak memory
original (PyTorch bf16, MPS) - 0.0 % / 0.0 % 0.968 / 0.950 2.94 / 2.36 28.9 -
bf16/ 16 GB not checkable on 24 GB (swaps); weights bit-identical to the original; 1 clip: 0.0 %, sim 0.985, DNSMOS 3.25 ~28 (swapping) -
fp16/ 16 GB not checkable on 24 GB; weights within 3e-8 of the original - -
8bit/ 8.5 GB 0.0 % / 0.0 % 0.970 / 0.955 3.11 / 2.72 5.9 17.2 GB
6bit/ 6.6 GB 0.0 % / 0.0 % 0.961 / 0.942 3.09 / 2.52 2.4 15.1 GB
4bit/ (experimental) 4.6 GB 3.6 % / 1.3 % 0.967 / 0.943 3.14 / 2.79 1.5 13.0 GB

4bit/ is below the threshold: it garbled one English tongue twister ("the shells she sells") and one Chinese word. Real-time factor = generation time / audio length, one sentence at a time; peak memory includes the audio tokenizer. bf16 / fp16 need more than 24 GB of unified memory.

How these were made

From MOSS-TTS v1.5 at revision cdd3b911b1: the original weights loaded in MLX and stored as bf16 or fp16, or quantized with MLX's affine quantization (8bit, 6bit, 4bit, group size 64; layers MLX can't quantize, such as convolutions and norms, stay bf16). Every build was checked against the original before upload; the sha256 column above is what the files must hash to.

License

Apache-2.0. These files are a derivative of the original model and carry its license: it is copied in this repository, and it governs your use of these files. The original model card follows unchanged.

Original model card

MOSS-TTS Family


    

MOSS-TTS-v1.5

MOSS-TTS-v1.5 is continued from MOSS-TTS 1.0. It preserves the main 1.0 capabilities, including zero-shot voice cloning, long-form speech generation, token-level duration control, Pinyin/IPA pronunciation control, multilingual synthesis, and code-switching. For the full 1.0 feature walkthrough, input schema, decoding hyperparameters, and evaluation tables, please refer to the MOSS-TTS 1.0 README.

Compared with MOSS-TTS 1.0, v1.5 focuses on the following improvements:

  • Stronger multilingual synthesis with language tags: when the language field is omitted, v1.5 may improve some languages and regress slightly on others compared with 1.0. When the language is specified, v1.5 is stronger than 1.0 on almost all supported languages. Set the tag when building the user message, for example processor.build_user_message(text=text_fr, language="French").
  • More stable voice cloning: v1.5 improves speaker similarity and reduces cloning variance, making repeated generations more consistent.
  • Better long-reference, short-text cloning: v1.5 handles scenarios where the reference audio is much longer than the target text more reliably than 1.0.
  • More stable punctuation-following prosody: v1.5 follows punctuation-driven pauses more closely, especially in long sentences.
  • Explicit pause control: v1.5 supports inline pause markers such as "[pause 3.2s]". For example, 我今天学习了一首中国的古诗，它的名字是[pause 3.2s]静夜思！ inserts an explicit 3.2s pause before 静夜思.

Supported Languages

MOSS-TTS-v1.5 currently supports 31 languages. It keeps the 20 languages supported by MOSS-TTS 1.0 and extends multilingual continued training to additional languages including Cantonese, Dutch, Finnish, Hindi, Macedonian, Malay, Romanian, Swahili, Tagalog, Thai, and Vietnamese.

Language Code Flag Language Code Flag Language Code Flag
Chinese zh 🇨🇳 Cantonese yue 🇭🇰 English en 🇺🇸
Arabic ar 🇸🇦 Czech cs 🇨🇿 Danish da 🇩🇰
Dutch nl 🇳🇱 Finnish fi 🇫🇮 French fr 🇫🇷
German de 🇩🇪 Greek el 🇬🇷 Hebrew he 🇮🇱
Hindi hi 🇮🇳 Hungarian hu 🇭🇺 Italian it 🇮🇹
Japanese ja 🇯🇵 Korean ko 🇰🇷 Macedonian mk 🇲🇰
Malay ms 🇲🇾 Persian (Farsi) fa 🇮🇷 Polish pl 🇵🇱
Portuguese pt 🇵🇹 Romanian ro 🇷🇴 Russian ru 🇷🇺
Spanish es 🇪🇸 Swahili sw 🇹🇿 Swedish sv 🇸🇪
Tagalog tl 🇵🇭 Thai th 🇹🇭 Turkish tr 🇹🇷
Vietnamese vi 🇻🇳

Quick Start

Environment Setup

We recommend a clean, isolated Python environment with Transformers 5.0.0 to avoid dependency conflicts.

conda create -n moss-tts python=3.12 -y
conda activate moss-tts

Install all required dependencies:

git clone https://github.com/OpenMOSS/MOSS-TTS.git
cd MOSS-TTS
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e .

(Optional) Install FlashAttention 2

For better speed and lower GPU memory usage, you can install FlashAttention 2 if your hardware supports it.

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"

If your machine has limited RAM and many CPU cores, you can cap build parallelism:

MAX_JOBS=4 pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[flash-attn]"

Notes:

  • Dependencies are managed in pyproject.toml, which currently pins torch==2.9.1+cu128 and torchaudio==2.9.1+cu128.
  • If FlashAttention 2 fails to build on your machine, you can skip it and use the default attention backend.
  • FlashAttention 2 is only available on supported GPUs and is typically used with torch.float16 or torch.bfloat16.

Basic Usage

Tip: MOSS-TTS-v1.5 uses the same generation API as the 1.0 MossTTSDelay-8B checkpoint. For multilingual inputs, set language whenever the language is known.

MOSS-TTS provides a convenient generate interface for rapid usage. The examples below cover:

  1. Direct generation (Chinese / English / multilingual text with language tags / Pinyin / IPA)
  2. Voice cloning
  3. Duration control
  4. Explicit pause control with [pause X.Ys]
from pathlib import Path
import importlib.util
import torch
import torchaudio
from transformers import AutoModel, AutoProcessor
# Disable the broken cuDNN SDPA backend
torch.backends.cuda.enable_cudnn_sdp(False)
# Keep these enabled as fallbacks
torch.backends.cuda.enable_flash_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(True)
torch.backends.cuda.enable_math_sdp(True)


pretrained_model_name_or_path = "OpenMOSS-Team/MOSS-TTS-v1.5"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

def resolve_attn_implementation() -> str:
    # Prefer FlashAttention 2 when package + device conditions are met.
    if (
        device == "cuda"
        and importlib.util.find_spec("flash_attn") is not None
        and dtype in {torch.float16, torch.bfloat16}
    ):
        major, _ = torch.cuda.get_device_capability()
        if major >= 8:
            return "flash_attention_2"

    # CUDA fallback: use PyTorch SDPA kernels.
    if device == "cuda":
        return "sdpa"

    # CPU fallback.
    return "eager"


attn_implementation = resolve_attn_implementation()
print(f"[INFO] Using attn_implementation={attn_implementation}")

processor = AutoProcessor.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True,
)
processor.audio_tokenizer = processor.audio_tokenizer.to(device)

text_1 = "亲爱的你，\n你好呀。\n\n今天，我想用最认真、最温柔的声音，对你说一些重要的话。\n这些话，像一颗小小的星星，希望能在你的心里慢慢发光。\n\n首先，我想祝你——\n每天都能平平安安、快快乐乐。\n\n希望你早上醒来的时候，\n窗外有光，屋子里很安静，\n你的心是轻轻的，没有着急，也没有害怕。\n\n希望你吃饭的时候胃口很好，\n走路的时候脚步稳稳，\n晚上睡觉的时候，能做一个又一个甜甜的梦。\n\n我希望你能一直保持好奇心。\n对世界充满问题，\n对天空、星星、花草、书本和故事感兴趣。\n当你问“为什么”的时候，\n希望总有人愿意认真地听你说话。\n\n我也希望你学会温柔。\n温柔地对待朋友，\n温柔地对待小动物，\n也温柔地对待自己。\n\n如果有一天你犯了错，\n请不要太快责怪自己，\n因为每一个认真成长的人，\n都会在路上慢慢学会更好的方法。\n\n愿你拥有勇气。\n当你站在陌生的地方时，\n当你第一次举手发言时，\n当你遇到困难、感到害怕的时候，\n希望你能轻轻地告诉自己：\n“我可以试一试。”\n\n就算没有一次成功，也没有关系。\n失败不是坏事，\n它只是告诉你，你正在努力。\n\n我希望你学会分享快乐。\n把开心的事情告诉别人，\n把笑声送给身边的人，\n因为快乐被分享的时候，\n会变得更大、更亮。\n\n如果有一天你感到难过，\n我希望你知道——\n难过并不丢脸，\n哭泣也不是软弱。\n\n愿你能找到一个安全的地方，\n慢慢把心里的话说出来，\n然后再一次抬起头，看见希望。\n\n我还希望你能拥有梦想。\n这个梦想也许很大，\n也许很小，\n也许现在还说不清楚。\n\n没关系。\n梦想会和你一起长大，\n在时间里慢慢变得清楚。\n\n最后，我想送你一个最最重要的祝福：\n\n愿你被世界温柔对待，\n也愿你成为一个温柔的人。\n\n愿你的每一天，\n都值得被记住，\n都值得被珍惜。\n\n亲爱的你，\n请记住，\n你是独一无二的，\n你已经很棒了，\n而你的未来，\n一定会慢慢变得闪闪发光。\n\n祝你健康、勇敢、幸福，\n祝你永远带着笑容向前走。"
text_2 = "We stand on the threshold of the AI era.\nArtificial intelligence is no longer just a concept in laboratories, but is entering every industry, every creative endeavor, and every decision. It has learned to see, hear, speak, and think, and is beginning to become an extension of human capabilities. AI is not about replacing humans, but about amplifying human creativity, making knowledge more equitable, more efficient, and allowing imagination to reach further. A new era, jointly shaped by humans and intelligent systems, has arrived."
text_3 = "nin2 hao3，qing3 wen4 nin2 lai2 zi4 na3 zuo4 cheng2 shi4？"
text_4 = "nin2 hao3，qing4 wen3 nin2 lai2 zi4 na4 zuo3 cheng4 shi3？"
text_5 = "您好，请问您来自哪 zuo4 cheng2 shi4？"
text_6 = "/həloʊ, meɪ aɪ æsk wɪtʃ sɪti juː ɑːr frʌm?/"
text_7 = "Bonjour, je voudrais essayer une voix française naturelle et stable."
text_8 = "我今天学习了一首中国的古诗，它的名字是[pause 3.2s]静夜思！"

# Use audio from ./assets/audio to avoid downloading from the cloud.
ref_audio_1 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_zh.wav"
ref_audio_2 = "https://speech-demo.oss-cn-shanghai.aliyuncs.com/moss_tts_demo/tts_readme_demo/reference_en.m4a"

conversations = [
    # Direct TTS (no reference). Language tags are recommended in v1.5.
    [processor.build_user_message(text=text_1)],
    [processor.build_user_message(text=text_2)],
    # Direct TTS (no reference). For languages ​​other than Chinese and English, it is recommended to use language tags.
    [processor.build_user_message(text=text_7, language="French")],
    # Pinyin or IPA input
    [processor.build_user_message(text=text_3)],
    [processor.build_user_message(text=text_4)],
    [processor.build_user_message(text=text_5)],
    [processor.build_user_message(text=text_6)],
    # Explicit pause control. Use [pause X.Ys], such as [pause 3.2s].
    [processor.build_user_message(text=text_8)],
    # Voice cloning (with reference)
    [processor.build_user_message(text=text_1, reference=[ref_audio_1])],
    [processor.build_user_message(text=text_2, reference=[ref_audio_2])],
    # Duration control
    [processor.build_user_message(text=text_2, tokens=325)],
    [processor.build_user_message(text=text_2, tokens=600)],
]

model = AutoModel.from_pretrained(
    pretrained_model_name_or_path,
    trust_remote_code=True,
    # If FlashAttention 2 is installed, you can set attn_implementation="flash_attention_2"
    attn_implementation=attn_implementation,
    torch_dtype=dtype,
).to(device)
model.eval()

batch_size = 1

save_dir = Path("inference_root")
save_dir.mkdir(exist_ok=True, parents=True)
sample_idx = 0
with torch.no_grad():
    for start in range(0, len(conversations), batch_size):
        batch_conversations = conversations[start : start + batch_size]
        batch = processor(batch_conversations, mode="generation")
        input_ids = batch["input_ids"].to(device)
        attention_mask = batch["attention_mask"].to(device)

        outputs = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=4096,
        )

        for message in processor.decode(outputs):
            audio = message.audio_codes_list[0]
            out_path = save_dir / f"sample{sample_idx}.wav"
            sample_idx += 1
            torchaudio.save(out_path, audio.unsqueeze(0), processor.model_config.sampling_rate)

More Usage

MOSS-TTS-v1.5 is API-compatible with MOSS-TTS 1.0. For continuation with prefix audio, detailed UserMessage and AssistantMessage fields, generation hyperparameters, Pinyin/IPA preprocessing examples, and evaluation results, see the MOSS-TTS 1.0 README.

Downloads last month

-

Downloads are not tracked for this model. How to track
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
Text-to-Speech
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for moonbox-dev/MOSS-TTS-v1.5-MLX

Base model

OpenMOSS-Team/MOSS-TTS-v1.5
Quantized
(9)
this model

Paper for moonbox-dev/MOSS-TTS-v1.5-MLX