File size: 9,163 Bytes
550ce03 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 | # FP32 GPU inference and batching
The batched ONNX path uses **no PyTorch, FP16, TF32, or weight quantization**.
It reuses the released trained weights without retraining. The browser demo keeps
using the original single-request WASM model; these additional graphs are for
native CPU/CUDA deployments.
## Measured results
Test hardware: **NVIDIA GeForce RTX 5090, 32 GB**, driver 580.178.04, Linux,
Python 3.12.3, ONNX Runtime GPU 1.32.0 with CUDA 13 libraries. The environment
contained no PyTorch installation. **These are not RTX PRO 4000 measurements.**
Batch latency includes pronunciation, CPU/GPU transfers, synthesis and audio
returned to CPU memory. Models were loaded and warmed up first. Each row has
20 measured repeats, using two passages and both voices, repeated to fill the batch. Speaking speeds were calibrated
to approximately five or ten seconds per clip; actual lengths are in the report.
| Audio length per request | Batch size | Median batch latency | p95 batch latency |
|---|---:|---:|---:|
| ~5 seconds | 1 | 23.5 ms | 23.6 ms |
| ~5 seconds | 8 | 32.7 ms | 33.0 ms |
| ~5 seconds | 16 | 44.2 ms | 44.4 ms |
| ~5 seconds | 32 | 69.7 ms | 70.0 ms |
| ~10 seconds | 1 | 26.2 ms | 26.3 ms |
| ~10 seconds | 8 | 47.9 ms | 48.2 ms |
| ~10 seconds | 16 | 73.3 ms | 74.5 ms |
| ~10 seconds | 32 | 157.4 ms | 161.2 ms |
The 32-request, ten-second case was automatically split into two decoder groups
by the default padded-frame memory budget. With 16 ten-second requests, the new
batch path took about **73 ms**, compared with about **1,269 ms** to run those
requests serially through the original CUDA model: approximately **17.3×** faster
for that workload. The original comparison used the same precision and texts.
The microbatch queue was also tested with **64 requests submitted together**,
a maximum collection size of 32 and a 2 ms collection window:
| Clip length | All 64 requests completed | Completed within 0.5 seconds |
|---|---:|---:|
| ~5 seconds | 157 ms | 64 / 64 |
| ~10 seconds | 343 ms | 64 / 64 |
These burst results are a single controlled burst per clip-length setting,
not a production SLA or a maximum-capacity measurement. WAV encoding, HTTP,
network transfer, cold starts and unrelated GPU workloads are excluded. Larger
or more varied inputs and a longer queue change completion times. Throughput
and simultaneous-request deadlines must not be treated as interchangeable.
The largest sampled per-process GPU residency after a benchmark run was
**4,106 MiB**, including the comparison model's session. This is not a sampled
peak, a minimum VRAM requirement, or a guarantee for other batch/frame limits.
Test on your own RTX PRO 4000 before promising the same latency.
Raw data: [GPU benchmark](evaluation/gpu_batch_benchmark.json),
[GPU parity](evaluation/batched_validation_cuda.json),
[CPU parity](evaluation/batched_validation.json).
## Install and generate a batch
Use a fresh Python environment with an NVIDIA driver compatible with CUDA 13.
The GPU requirements install CUDA/cuDNN runtime libraries through pip; they do
not install a training framework. Do not install both `onnxruntime` and
`onnxruntime-gpu` in the same environment. The repository's training-package
installation is unnecessary for this standalone inference path.
```bash
git clone https://github.com/ub1979/little-tts
cd little-tts
python -m venv .venv-gpu
source .venv-gpu/bin/activate
pip install -r requirements-onnx-gpu.txt
python batched_onnx.py --device cuda --batch-size 16 \
--text "The morning sunlight streamed through the window, warming the quiet room." \
--text "The garden was quiet, and the air was filled with the scent of roses." \
--output batch-audio
```
To deploy only the GPU inference files from Hugging Face, rather than clone
the training repository, use the Hub CLI:
```bash
pip install 'huggingface-hub>=1.16,<2'
hf download TuringLogic/little-tts --local-dir little-tts-gpu \
--include 'onnx/batched/*' 'phonemizer/*' 'batched_onnx.py' 'batch_queue.py' \
'requirements-onnx-gpu.txt' 'LICENSE*' 'NOTICE'
```
The original CPU/browser graph is still `onnx/model.onnx`. The new graphs are
`onnx/batched/encoder.onnx` and `onnx/batched/decoder.onnx`, with metadata in
`onnx/batched/config.json`. All learned floating-point weights and audio outputs remain FP32.
```python
from batched_onnx import BatchedSynthesizer
synth = BatchedSynthesizer(".", device="cuda", max_batch=16,
max_padded_frames=16000)
results = synth.speak_batch(
["The garden was quiet, and the air was filled with roses.",
"The road curved gently through the hills."],
voices=["92", "6097"],
speeds=[1.0, 1.0],
seeds=[0, 1],
)
for waveform, durations in results:
print(len(waveform) / synth.sample_rate)
print(synth.last_stats)
```
Results are restored to input order, even when requests are length-sorted.
Voice 92 remains the recommended default. `--device cuda` fails if CUDA cannot
initialize; it does not silently switch the full session to CPU. Small shape
operations can still be assigned to CPU by ONNX Runtime.
## Combine independent callers
`speak_batch()` processes a supplied list. For independent concurrent callers,
use the bounded queue to collect requests into actual shared batches:
```python
from batch_queue import BatchQueue
with BatchQueue(synth, max_wait_ms=2, max_pending=256) as queue:
first = queue.submit("The morning was quiet, and the sky was clear.")
second = queue.submit("The garden was filled with the scent of roses.")
first_wave, _ = first.result()
second_wave, _ = second.result()
```
Use one queue per synthesizer. Direct concurrent calls to `speak_batch()` are
serialized to protect shared runtime state; the queue is what combines them.
Requests receive results at batch completion. Queue capacity is bounded, and
overload raises an error rather than accumulating unlimited work. Per-call
settings are validated; inference failures are propagated to the affected batch.
`max_batch` controls collection size. `max_padded_frames` limits
`batch_size × longest_predicted_frame_count` before allocating DSP tensors; an
oversized group is split automatically. This is a memory-control heuristic,
not an exact VRAM allocation limit. Each request is limited to 512 phoneme
tokens and 3,000 frames; split longer input into passages. For low latency with
ten-second clips, start with `max_batch=16` and measure the actual workload.
## What changed
- Real dynamic batches, token masks and frame masks replace batch-size-one
assumptions. Each request keeps its own voice, speed, pitch controls, noise
seed, durations and output length; padding cannot leak into real audio.
- Text encoding and waveform decoding are separate graphs. Only small duration
arrays return to the host to select the frame extent. I/O binding keeps the
large encoder hidden tensor on CUDA between stages.
- RoPE positions and other shape inputs come from cached host-defined ranges,
avoiding duration-dependent GPU-to-host shape discovery inside a monolithic
graph. Runtime graph optimization is enabled.
- The harmonic oscillator is evaluated in 16-harmonic tiles to bound temporary
memory. The noise filter uses the original real-valued DFT matrices as dense
matrix multiplications and four-way overlap-add, avoiding the slow DSP
convolution/transposed-convolution route measured on the test GPU.
- CUDA uses the NC1D convolution layout, and TF32 is explicitly disabled.
No TensorRT, PyTorch or quantization is required.
## Rebuild and validate without PyTorch
The checked-in graphs are ready to use. To rebuild them, install the small
additional graph-building requirement:
```bash
pip install -r requirements-onnx-build.txt
python scripts/build_batched_onnx.py
python scripts/validate_batched_onnx.py --device cuda \
--output evaluation/my_gpu_validation.json
```
The builder extracts unchanged learned tensors directly from `onnx/model.onnx`,
not from a PyTorch checkpoint. It records that source file's SHA256. The parity
checker exposes the original ONNX graph's noise input and supplies identical
noise to compare waveform samples. CPU and GPU checks cover both voices,
mixed lengths/speeds/pitch settings, input-order restoration, batch invariance
and forced memory-budget splitting. All tested token durations and output
lengths matched. Maximum GPU waveform error was approximately **0.1% of reference
peak amplitude**; this numerical check is not a human naturalness rating.
To reproduce the latency and simultaneous-burst tests:
```bash
python scripts/benchmark_batched_onnx.py --device cuda \
--batch-sizes 1 2 4 8 16 32 --seconds 5 10 \
--repeats 20 --serial-repeats 3 --burst 64 \
--output evaluation/my_gpu_benchmark.json
```
For CPU-only validation, use `requirements-onnx.txt` and
`requirements-onnx-build.txt` in a separate environment, then pass `--device cpu`.
The existing browser demo does not use or download the batched graphs.
Weights, including these derived graphs, are CC BY-NC 4.0. Source code is
Apache-2.0; third-party pronunciation assets retain their included notices.
|