Update README.md
Browse files
README.md
CHANGED
|
@@ -13,115 +13,7 @@ tags:
|
|
| 13 |
- onnxruntime
|
| 14 |
---
|
| 15 |
|
| 16 |
-
#
|
| 17 |
-
|
| 18 |
-
ONNX export of [Pocket TTS](https://huggingface.co/kyutai/pocket-tts) for lightweight text-to-speech with zero-shot voice cloning.
|
| 19 |
-
|
| 20 |
-
## Model Description
|
| 21 |
-
|
| 22 |
-
Pocket TTS is a compact text-to-speech model from Kyutai that supports voice cloning from short audio samples. This ONNX export provides:
|
| 23 |
-
|
| 24 |
-
- **Zero-shot voice cloning** from any audio reference
|
| 25 |
-
- **INT8 quantized models** for fast CPU inference
|
| 26 |
-
- **Streaming support** with adaptive chunking for real-time playback
|
| 27 |
-
- **Temperature control** for generation diversity
|
| 28 |
-
- **Dual model architecture** for flexible flow matching
|
| 29 |
-
|
| 30 |
-
## Architecture
|
| 31 |
-
|
| 32 |
-
This export uses a **dual model split** for the Flow LM:
|
| 33 |
-
|
| 34 |
-
```
|
| 35 |
-
flow_lm_main - Transformer/conditioner (produces conditioning vectors)
|
| 36 |
-
flow_lm_flow - Flow network only (Euler integration for latent sampling)
|
| 37 |
-
```
|
| 38 |
-
|
| 39 |
-
This architecture enables:
|
| 40 |
-
- **Temperature control**: Adjust generation diversity via noise scaling
|
| 41 |
-
- **Variable LSD steps**: Trade off speed vs quality
|
| 42 |
-
- **External flow loop**: Full control over the sampling process
|
| 43 |
-
|
| 44 |
-
## Performance
|
| 45 |
-
|
| 46 |
-
Benchmarked on a 16-core CPU, `lsd_steps=1`, measuring flow loop + mimi decoder (excluding voice encoder).
|
| 47 |
-
|
| 48 |
-
| Precision | Flow Main | Flow Net | Decoder | Total Size | RTFx (CPU) |
|
| 49 |
-
|-----------|-----------|----------|---------|------------|------------|
|
| 50 |
-
| INT8 | 76 MB | 10 MB | 23 MB | ~200 MB | **~4.0x** |
|
| 51 |
-
| FP32 | 303 MB | 39 MB | 42 MB | ~475 MB | **~2.8x** |
|
| 52 |
-
| PyTorch FP32 (reference) | — | — | — | — | ~4.0x |
|
| 53 |
-
|
| 54 |
-
RTFx = Real-time factor (>1.0 means faster than real-time). ONNX INT8 matches PyTorch FP32 throughput.
|
| 55 |
-
|
| 56 |
-
### Optimized Inference
|
| 57 |
-
|
| 58 |
-
Thread count is automatically tuned to avoid over-subscription on the small sequential matmuls in the autoregressive loop (`intra_op_num_threads=min(cpu_count, 4)`, `inter_op_num_threads=1`). This alone provides a **~2x speedup** over default ORT settings on multi-core machines.
|
| 59 |
-
|
| 60 |
-
### Decoding Strategy
|
| 61 |
-
|
| 62 |
-
- **Offline (`generate`)**: Uses **threaded parallel decoding** — the mimi decoder runs in a background thread, decoding 12-frame chunks while the flow loop generates the next frames. This overlaps generation and decoding for maximum throughput.
|
| 63 |
-
- **Streaming (`stream`)**: Uses **adaptive chunking** (starts at 2 frames). This ensures instant start (low TTFB) while scaling up chunk sizes for throughput.
|
| 64 |
-
|
| 65 |
-
## Usage
|
| 66 |
-
|
| 67 |
-
```python
|
| 68 |
-
from pocket_tts_onnx import PocketTTSOnnx
|
| 69 |
-
|
| 70 |
-
# Load model (INT8 by default)
|
| 71 |
-
tts = PocketTTSOnnx()
|
| 72 |
-
|
| 73 |
-
# Generate speech with voice cloning
|
| 74 |
-
audio = tts.generate(
|
| 75 |
-
text="Hello, this is a test of voice cloning.",
|
| 76 |
-
voice="reference_sample.wav"
|
| 77 |
-
)
|
| 78 |
-
|
| 79 |
-
# Save output
|
| 80 |
-
tts.save_audio(audio, "output.wav")
|
| 81 |
-
```
|
| 82 |
-
|
| 83 |
-
### Temperature Control
|
| 84 |
-
|
| 85 |
-
Adjust generation diversity with the `temperature` parameter:
|
| 86 |
-
|
| 87 |
-
```python
|
| 88 |
-
# More deterministic (lower temperature)
|
| 89 |
-
tts = PocketTTSOnnx(temperature=0.3)
|
| 90 |
-
|
| 91 |
-
# Default balance
|
| 92 |
-
tts = PocketTTSOnnx(temperature=0.7)
|
| 93 |
-
|
| 94 |
-
# More diverse/expressive (higher temperature)
|
| 95 |
-
tts = PocketTTSOnnx(temperature=1.0)
|
| 96 |
-
```
|
| 97 |
-
|
| 98 |
-
### LSD Steps
|
| 99 |
-
|
| 100 |
-
Trade off speed vs quality with `lsd_steps`:
|
| 101 |
-
|
| 102 |
-
```python
|
| 103 |
-
# Default (10 steps)
|
| 104 |
-
tts = PocketTTSOnnx(lsd_steps=10)
|
| 105 |
-
|
| 106 |
-
# Faster (fewer steps, lower quality)
|
| 107 |
-
tts = PocketTTSOnnx(lsd_steps=1)
|
| 108 |
-
```
|
| 109 |
-
|
| 110 |
-
### Streaming Mode
|
| 111 |
-
|
| 112 |
-
For real-time applications with low time-to-first-audio:
|
| 113 |
-
|
| 114 |
-
```python
|
| 115 |
-
for chunk in tts.stream("Hello world!", voice="reference_sample.wav"):
|
| 116 |
-
play_audio(chunk) # Process each chunk as it arrives
|
| 117 |
-
```
|
| 118 |
-
|
| 119 |
-
### Command Line
|
| 120 |
-
|
| 121 |
-
```bash
|
| 122 |
-
python generate.py "Hello, this is a test." reference_sample.wav output.wav
|
| 123 |
-
```
|
| 124 |
-
|
| 125 |
## Files
|
| 126 |
|
| 127 |
```
|
|
@@ -142,31 +34,3 @@ pocket-tts-onnx/
|
|
| 142 |
├── requirements.txt # Python dependencies
|
| 143 |
└── README.md
|
| 144 |
```
|
| 145 |
-
|
| 146 |
-
## Requirements
|
| 147 |
-
|
| 148 |
-
```
|
| 149 |
-
onnxruntime>=1.16.0
|
| 150 |
-
numpy
|
| 151 |
-
soundfile
|
| 152 |
-
sentencepiece
|
| 153 |
-
scipy # Only needed if resampling from non-24kHz audio
|
| 154 |
-
```
|
| 155 |
-
|
| 156 |
-
Install with:
|
| 157 |
-
```bash
|
| 158 |
-
pip install -r requirements.txt
|
| 159 |
-
```
|
| 160 |
-
|
| 161 |
-
## License
|
| 162 |
-
|
| 163 |
-
- **Models**: CC BY 4.0 (inherited from [kyutai/pocket-tts](https://huggingface.co/kyutai/pocket-tts))
|
| 164 |
-
- **Code**: Apache 2.0
|
| 165 |
-
|
| 166 |
-
## Prohibited Use
|
| 167 |
-
|
| 168 |
-
Use of our model must comply with all applicable laws and regulations and must not result in, involve, or facilitate any illegal, harmful, deceptive, fraudulent, or unauthorized activity. Prohibited uses include, without limitation, voice impersonation or cloning without explicit and lawful consent; misinformation, disinformation, or deception (including fake news, fraudulent calls, or presenting generated content as genuine recordings of real people or events); and the generation of unlawful, harmful, libelous, abusive, harassing, discriminatory, hateful, or privacy-invasive content. We disclaim all liability for any non-compliant use.
|
| 169 |
-
|
| 170 |
-
## Acknowledgments
|
| 171 |
-
|
| 172 |
-
- [Kyutai](https://kyutai.org/) for the original Pocket TTS model
|
|
|
|
| 13 |
- onnxruntime
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# Voice Clone Pro ONNX
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
## Files
|
| 18 |
|
| 19 |
```
|
|
|
|
| 34 |
├── requirements.txt # Python dependencies
|
| 35 |
└── README.md
|
| 36 |
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|