| # Benchmark | |
| Full pipeline (frontend β AR β cond β DiT β vocoder), same text / reference audio / seed. | |
| Times in milliseconds, median of three warm runs (rows 2β6). Rows 2β6 were re-measured on | |
| 2026-11-03 with the current development build (ggml-only F32 HiFi-GAN vocoder with sub-pixel upsampling, | |
| RoPE/adaLN/mish fusion included); rows 1 and 4 are the unchanged torch reference. | |
| | # | configuration | total | G2P/frontend | cond | AR | DiT | vocoder | cache | | |
| |---|---|---|---|---|---|---|---|---| | |
| | 1 | v5turbo torch | **3142** | 162 | 251 | ~2103* | 561 | 65 | β | | |
| | 2 | v5turbo ggml (AR fp16) | **945** | 102 | 93 | 363 | 275 | 68 | β | | |
| | 3 | v5turbo ggml (AR q8) | **1028** | 101 | 95 | 425 | 279 | 67 | β | | |
| | 4 | v5dev torch | **11716** | 163 | 247 | ~2326* | 8924 | 56 | β | | |
| | 5 | v5dev ggml | **5415** | 102 | 126 | 456 | 4617 | 68 | β | | |
| | 6 | v5dev ggml + cache-dit | **4740** | 101 | 120 | 422 | 4005 | 69 | **17/64 = 26.6%** | | |
| The vocoder is the ggml F32 HiFi-GAN with **sub-pixel upsampling**: the five transposed | |
| convolutions are replaced by one 3-tap conv1d on the input grid plus a phase interleave | |
| (`src/gsv_vocoder_ggml.cpp::upsample_subpixel`), and every convolution runs through | |
| `ggml_conv_direct_1d_fused` (no im2col buffer; bias/LeakyReLU, the resblock input LeakyReLU and | |
| the residual add all fold into the same kernel). For the 3.4 s clip in this table the vocoder | |
| takes **~68 ms**, down from ~140 ms with `ggml_conv_transpose_1d`, ~115 ms with sub-pixel + im2col | |
| and ~75 ms with per-conv direct conv (-51% overall). The old ONNX vocoder was ~114 ms. AR varies | |
| by roughly Β±20% between runs on this machine (CPU sidecar + GPU sharing), which dominates the | |
| spread of the totals. | |
| \* torch's AR loop is captured as a single CUDA graph and cannot be timed separately; this | |
| column is `total β (frontend + cond + DiT + vocoder)` (AR + scheduling overhead). | |
| ## Environment | |
| - CPU: Intel Xeon E5-2675 v3 @ 1.80 GHz (16 threads) | |
| - GPU: NVIDIA GeForce RTX 2070 8 GB, driver supporting Vulkan 1.2 | |
| - OS: Windows 10 x64 | |
| - ggml: Vulkan backend (`--device vulkan`), weights as described per row | |
| (AR fp16 = `gsv-ar-f16.gguf`, AR q8 = `gsv-ar-q8.gguf`; cond/DiT F16; refcode F16) | |
| - vocoder: ggml HiFi-GAN, F32 weights (`gsv-vocoder-f32.gguf`), Vulkan backend | |
| - torch: 2.12.1 + CUDA 13.0, fp16, `use_cuda_graph=True`, batch size 1, text split `cut0` | |
| - sample steps: v5turbo 4 / cfg 0.0; v5dev 32 / cfg 1.30 | |
| - cache-dit: DBCache `Fn=8`, threshold 0.02, warmup 8, downsample 4 | |
| Notes: | |
| - Text: `δ½ ε₯½οΌζ¬’θΏδ½Ώη¨θ―ι³εζη³»η»γ` | |
| - Reference: 16 kHz Chinese prompt, ~3.4 s output audio in every row. | |
| - Timings vary by a few percent between runs; the torch numbers include CUDA graph replay | |
| (two warm-up runs with the same text are done before the measured run). | |
| ## DiT fusion (RoPE + adaLN) | |
| `gsv_dit.cpp` builds the DiT step graph with a fused RoPE (`ggml_rope_ext`, head 0 only, with a | |
| `pos` tensor and freq factors that reproduce the checkpoint `inv_freq`) and a fused | |
| `LN(x)Β·(1+scale)+shift` (`ggml_layernorm_affine` with `wb = [1+scale, shift]`), replacing the | |
| per-block cos/sin multiply chain and the `norm β add β mul β add` sequence. | |
| Same machine, turbo-q8, same text / seed, warm: | |
| | DiT | before | after | delta | | |
| |---|---|---|---| | |
| | v5turbo DiT | 333 ms | 291 ms | β12.6% | | |
| | v5turbo total | 1130 ms | 1033 ms | β8.6% | | |
| Audio correlation against the pre-fusion build is 0.9955; two runs with the same seed are | |
| bit-identical (corr 1.000000), so determinism is preserved. The `mish` epilogue fusion (A4 in the | |
| DiT report) is also in: `ggml_add_act` gained an `act=3` (mish) branch on CPU and in the Vulkan | |
| shader, and `conv_pos` folds bias + mish into one node. Its effect is below the 0.2% noise floor, | |
| so the numbers above do not change. `tests/test_add_act.cpp` checks all four activations against | |
| a host reference on CPU and on every GPU backend; CI runs it under Mesa lavapipe, which exercises | |
| the Vulkan shader branch (max|diff| 2.4e-7). | |
| ## CPU backend threading | |
| ggml's barrier spins while waiting, so on SMT CPUs the default hardware_concurrency() (logical | |
| threads) oversubscribes and slows requests down. gsv_server now defaults to physical cores when | |
| --threads is not given (hardware_concurrency()/2 when > 8). | |
| Windows, Xeon E5-2675 v3 (16 cores / 32 threads, 1.8 GHz), turbo-q8, 4 steps, 3.04 s audio: | |
| | threads | total | DiT | cond | AR | vocoder | | |
| |---|---|---|---|---|---| | |
| | 32 (old default) | 39.9 s | 28.8 s | 1.80 s | 2.26 s | 3.07 s | | |
| | 16 (new default) | 35.4 s | 29.0 s | 1.20 s | 1.52 s | 1.62 s | | |
| | 8 | 35.9 s | 28.8 s | 1.15 s | 1.54 s | 1.93 s | | |
| DiT is memory-bound in this configuration and does not improve past 8 threads; the win is in the | |
| AR/cond/vocoder stages. --threads N overrides the default. | |
| ## What is not parallelised | |
| - **Pipeline overlap** (AR of segment i+1 while DiT/vocoder runs segment i) is not implemented: | |
| ggml exposes a single backend queue, so both stages would serialize on the device anyway, and | |
| on CPU they compete for the same cores. Batches are formed *across* requests instead | |
| (/batch, AR batcher, request scheduler). | |
| - **Larger conv tiles** in conv_direct_1d were not beneficial: the kernel already uses 6-wide | |
| time blocks x 16-wide output blocks, and DiT (the dominant CPU stage) is memory-bound, so | |
| thread count beyond 8 did not move it. | |