File size: 5,326 Bytes
e41361a 7251111 e41361a 7251111 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 | ---
license: apache-2.0
base_model:
- FunAudioLLM/Fun-CosyVoice3-0.5B-2512
pipeline_tag: text-to-speech
language:
- zh
- en
- ja
- ko
- de
- es
- fr
- it
- ru
tags:
- cosyvoice
- gguf
- streaming-tts
- text-to-speech
- cpp
- knowledge-distillation
---
# cosyvoice3_stream.cpp
Runtime assets for
[cosyvoice3_yq_cpp](https://gitee.com/yang-qi1222/cosyvoice3_yq_cpp), an
inference-only CosyVoice3 C++/GGML streaming runtime.
This repository contains one project-compressed GGUF model, the two ONNX models
needed to extract a voice prompt from reference audio, and ten small precomputed
prompt features for immediate testing. Source code is distributed separately on
Gitee.
## Files
| Path | Bytes | Purpose |
| --- | ---: | --- |
| `models/student12_mlp_inner_q8_flow4_hift_f16.gguf` | 1,637,038,400 | Student12 LLM + Flow4 + HiFT runtime model |
| `frontend/speech_tokenizer_v3.onnx` | 969,451,503 | Speech-token extraction from reference audio |
| `frontend/campplus.onnx` | 28,303,423 | Speaker embedding extraction |
| `prompts/control10_01.gguf` ... `control10_10.gguf` | 51,296-188,576 each | Ready-to-use synthetic evaluation voices |
Every binary is pinned in `SHA256SUMS`. The source repository also ships a
machine-readable manifest used by its downloader.
## Model Lineage
The base model is
[FunAudioLLM/Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512).
The published GGUF applies the following project-specific deployment changes:
- a 12-layer student speech LLM obtained through structured layer pruning and
distillation;
- Flow inference distilled from 10 steps to 4 steps;
- GGUF conversion with inner LLM MLP tensors quantized to Q8 and Flow/HiFT kept
in F16.
The LLM compression workflow was informed by
[SPADE](https://arxiv.org/abs/2509.20802), with additional material on the
[SPADE project page](https://mm.kaist.ac.kr/projects/SPADE/). This is an
independent CosyVoice3 engineering adaptation, not an official SPADE model or a
claim that the paper's reported metrics transfer to this model.
## Prompt Features
`control10_01` through `control10_10` are features extracted from synthetic
teacher-model outputs used in this project's five-male/five-female Control10
evaluation. No source WAV files are distributed, and these prompts are not
presented as the voice of any real person.
Use a prompt directly for the lowest startup overhead. To clone a new authorized
reference voice, use `speech_tokenizer_v3.onnx` and `campplus.onnx` with the
audio-enabled runtime to create a new `prompt_speech.gguf`.
## Download And Run
```bash
git clone https://gitee.com/yang-qi1222/cosyvoice3_yq_cpp.git
cd cosyvoice3_yq_cpp
python3 -m pip install -r requirements-tools.txt
python3 scripts/download_assets.py \
--manifest manifests/assets.example.json \
--asset-root assets
```
Build and start the feature-only CPU server:
```bash
scripts/build_runtime.sh \
--mode feature \
--backend cpu \
--build-dir build/feature-cpu
scripts/start_server.sh \
--build-dir build/feature-cpu \
--backend cpu \
--model assets/models/student12_mlp_inner_q8_flow4_hift_f16.gguf \
--prompt-speech assets/prompts/control10_01.gguf \
--voice control10_01 \
--port 8080
```
CUDA architecture, thread count, chunk size, and FlashAttention settings must be
selected and validated for the deployment machine. See the source repository for
the audio-input build and complete server examples.
## Measured Baseline
The accepted project baseline covers only an NVIDIA RTX 5880 Ada Generation GPU,
one serial request at a time, 16 host threads, `chunk_tokens=75`, LLM
FlashAttention disabled, and Flow FlashAttention enabled.
| Metric | Project result |
| --- | ---: |
| Control10 median RTF | 0.104273 |
| First speech token | about 9.1 ms |
| Median first PCM | about 223.2 ms |
| Natural EOS | 10/10 |
| Resident soak | 100/100 |
| Process RSS | about 1,181 MiB |
| Device-level GPU memory used | about 2,461 MiB |
These measurements must not be extrapolated to RTX 4060, Jetson AGX, other GPU
architectures, concurrent service, or long-context synthesis.
## Limitations
- This repository distributes inference assets, not training checkpoints or
training code.
- The reference-audio path also requires an ONNX Runtime C/C++ SDK at build time;
the two ONNX model files do not replace that dependency.
- Voice cloning must only be performed with appropriate speaker consent and data
rights.
- The project is community maintained and is not affiliated with or endorsed by
the official CosyVoice or SPADE teams.
## License And Acknowledgements
Model assets are distributed under Apache-2.0. The C++ source repository uses the
MIT license; source and model licenses are separate.
This work depends on and thanks:
- [QwenAudio/CosyVoice](https://github.com/QwenAudio/CosyVoice) and the
[CosyVoice3 paper](https://arxiv.org/abs/2505.17589);
- [Lourdle/cosyvoice.cpp](https://github.com/Lourdle/cosyvoice.cpp);
- [SPADE](https://arxiv.org/abs/2509.20802);
- [ggml](https://github.com/ggml-org/ggml),
[llama.cpp](https://github.com/ggml-org/llama.cpp), and
[ONNX Runtime](https://github.com/microsoft/onnxruntime).
Please cite the original CosyVoice3 and SPADE papers when this model is used in
research comparisons.
|