| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| - ja |
| - ko |
| - de |
| - fr |
| - es |
| - id |
| - it |
| - th |
| - pt |
| - ru |
| - ms |
| - vi |
| library_name: c4tts |
| pipeline_tag: text-to-speech |
| base_model: netease-youdao/Confucius4-TTS |
| tags: |
| - text-to-speech |
| - tts |
| - voice-cloning |
| - zero-shot |
| - multilingual |
| - cross-lingual |
| - apple-silicon |
| - mlx |
| - metal |
| - confucius4-tts |
| --- |
| |
| # c4tts weights — Confucius4-TTS for Apple Silicon (C++/MLX) |
|
|
| Runtime weights for **c4tts**, a standalone C++/MLX inference engine that runs |
| the [Confucius4-TTS](https://huggingface.co/netease-youdao/Confucius4-TTS) |
| multilingual, zero-shot, cross-lingual TTS model on Apple Silicon with **no |
| Python dependency at run time**. |
|
|
| This repository hosts the model converted into c4tts's on-disk format (a |
| directory of `.npy` tensors that MLX memory-maps, plus the SentencePiece vocab). |
| It is a **repackaging** of Confucius4-TTS and its component models — see |
| [Provenance & licensing](#provenance--licensing). It does **not** contain new |
| trained weights. |
|
|
| - **Architecture:** speech-encoder + LLM (text→semantic, T2S) → flow-matching |
| diffusion (semantic→mel, S2A) → BigVGAN vocoder (mel→waveform) |
| - **Output:** 22.05 kHz mono waveform |
| - **Languages:** 14 (zh, en, ja, ko, de, fr, es, id, it, th, pt, ru, ms, vi) |
| - **Capabilities:** zero-shot voice cloning from a short reference clip (no |
| transcript needed), cross-lingual voice transfer |
| - **Precision:** weights shipped as float32; the engine can run the T2S decoder |
| in int8 (`quantized_matmul`) at run time for ~2× speed at ~0.9999 logit cosine |
| - **Footprint:** ~4.8 GB (only the tensors the runtime actually loads — e.g. |
| W2V-BERT is trimmed to the layers used for semantic-feature extraction) |
|
|
| ## Intended use |
|
|
| Local, offline, multilingual TTS and zero-shot voice cloning on macOS / Apple |
| Silicon, via the c4tts engine — e.g. a CLI (`c4tts_cli synth`) or the built-in |
| HTTP server with an OpenAI-style `/v1/audio/speech` API and a web console. |
|
|
| **Not** intended for: real-time deepfakes, impersonation without consent, or any |
| use prohibited by the upstream licenses. |
|
|
| ## How to use |
|
|
| These are weights for the **c4tts** engine (a C++/MLX runtime), not a |
| 🤗 Transformers checkpoint — they won't load with `AutoModel`. |
|
|
| 1. Build the c4tts engine for Apple Silicon (CMake + MLX). |
| 2. Download this repo into the engine's `bin/` directory: |
| ```bash |
| huggingface-cli download <this-repo> --local-dir bin |
| ``` |
| Layout (8 files): `bin/{t2s,s2a,w2vbert,campplus,bigvgan,audio}.safetensors` + |
| `bin/tokenizer/vocab.tsv`. Each module is one memory-mapped safetensors file. |
| 3. Synthesize (text is tokenized in C++ — no Python at run time): |
| ```bash |
| # CLI: clone the voice in ref.wav and read the text |
| ./c4tts_cli synth --prompt ref.wav --lang zh --text "你好，世界。" --out out.wav |
| |
| # or serve an OpenAI-style API + web console |
| ./c4tts_cli --web --web-key <key> |
| ``` |
|
|
| Reference audio: a few seconds of clean mono speech is enough for zero-shot |
| cloning; pass `--lang` to select the synthesis language (cross-lingual transfer |
| is supported). |
|
|
| ## Model components |
|
|
| | Component | Role | Source | |
| | --- | --- | --- | |
| | `t2s/` | Text→semantic LLM (GPT-2-style, 24 layers) + speaker encoder | Confucius4-TTS `t2s_model.safetensors` | |
| | `s2a/` | Semantic→mel flow-matching DiT + WaveNet | Confucius4-TTS `s2a_model.pt` | |
| | `tokenizer/vocab.tsv` | LLaMA SentencePiece BPE tokenizer | Confucius4-TTS | |
| | `w2vbert/` | W2V-BERT 2.0 semantic-feature encoder (layers used for extraction only) | `facebook/w2v-bert-2.0` | |
| | `campplus/` | CAMPPlus speaker embedding | `funasr/campplus` (`campplus_cn_common.bin`) | |
| | `bigvgan/` | BigVGAN v2 neural vocoder (22 kHz, 80-band, 256×) | `nvidia/bigvgan_v2_22khz_80band_256x` | |
| | `audio/` | Baked mel-filterbank + Hann window constants | derived | |
|
|
| ## Provenance & licensing |
|
|
| This is a **derivative / repackaged** distribution. You must comply with the |
| license of **every** upstream component: |
|
|
| | Upstream | License | Link | |
| | --- | --- | --- | |
| | Confucius4-TTS (T2S, S2A, tokenizer) | Apache-2.0 | https://huggingface.co/netease-youdao/Confucius4-TTS | |
| | W2V-BERT 2.0 (semantic encoder) | MIT | https://huggingface.co/facebook/w2v-bert-2.0 | |
| | BigVGAN v2 (vocoder) | MIT | https://huggingface.co/nvidia/bigvgan_v2_22khz_80band_256x | |
| | CAMPPlus (speaker embedding) | see source | https://huggingface.co/funasr/campplus | |
|
|
| The `license: apache-2.0` tag reflects the primary model (Confucius4-TTS). |
| Verify the CAMPPlus license for your use case before redistribution. |
|
|
| ## Limitations & responsible use |
|
|
| - Quality varies with reference-audio quality and is best on the 14 supported |
| languages; out-of-distribution text/voices may degrade. |
| - **Voice cloning is powerful and easily misused.** Only clone voices you have |
| explicit consent to use. Do not use this model to impersonate real people, to |
| deceive, or to generate audio you are not authorized to produce. Consider |
| disclosing AI-generated audio and watermarking outputs. |
| - Synthesis is non-deterministic under sampling; greedy decoding is reproducible. |
|
|
| ## Citation |
|
|
| Please cite the upstream Confucius4-TTS work: |
|
|
| ```bibtex |
| @misc{confucius4tts, |
| title = {Confucius4-TTS: a Multilingual and Cross-Lingual Zero-Shot TTS Engine}, |
| author = {NetEase Youdao}, |
| url = {https://huggingface.co/netease-youdao/Confucius4-TTS} |
| } |
| ``` |
|
|
| *c4tts is an independent C++/MLX runtime; it is not affiliated with or endorsed |
| by NetEase Youdao, Meta, NVIDIA, or FunASR.* |
|
|
