# Benchmark

Full pipeline (frontend → AR → cond → DiT → vocoder), same text / reference audio / seed.
Times in milliseconds, median of three warm runs (rows 2–6). Rows 2–6 were re-measured on
2026-11-03 with the current development build (ggml-only F32 HiFi-GAN vocoder with sub-pixel upsampling,
RoPE/adaLN/mish fusion included); rows 1 and 4 are the unchanged torch reference.

| # | configuration | total | G2P/frontend | cond | AR | DiT | vocoder | cache |
|---|---|---|---|---|---|---|---|---|
| 1 | v5turbo torch | **3142** | 162 | 251 | ~2103* | 561 | 65 | — |
| 2 | v5turbo ggml (AR fp16) | **945** | 102 | 93 | 363 | 275 | 68 | — |
| 3 | v5turbo ggml (AR q8) | **1028** | 101 | 95 | 425 | 279 | 67 | — |
| 4 | v5dev torch | **11716** | 163 | 247 | ~2326* | 8924 | 56 | — |
| 5 | v5dev ggml | **5415** | 102 | 126 | 456 | 4617 | 68 | — |
| 6 | v5dev ggml + cache-dit | **4740** | 101 | 120 | 422 | 4005 | 69 | **17/64 = 26.6%** |

The vocoder is the ggml F32 HiFi-GAN with **sub-pixel upsampling**: the five transposed
convolutions are replaced by one 3-tap conv1d on the input grid plus a phase interleave
(`src/gsv_vocoder_ggml.cpp::upsample_subpixel`), and every convolution runs through
`ggml_conv_direct_1d_fused` (no im2col buffer; bias/LeakyReLU, the resblock input LeakyReLU and
the residual add all fold into the same kernel). For the 3.4 s clip in this table the vocoder
takes **~68 ms**, down from ~140 ms with `ggml_conv_transpose_1d`, ~115 ms with sub-pixel + im2col
and ~75 ms with per-conv direct conv (-51% overall). The old ONNX vocoder was ~114 ms. AR varies
by roughly ±20% between runs on this machine (CPU sidecar + GPU sharing), which dominates the
spread of the totals.

\* torch's AR loop is captured as a single CUDA graph and cannot be timed separately; this
column is `total − (frontend + cond + DiT + vocoder)` (AR + scheduling overhead).

## Environment

- CPU: Intel Xeon E5-2675 v3 @ 1.80 GHz (16 threads)
- GPU: NVIDIA GeForce RTX 2070 8 GB, driver supporting Vulkan 1.2
- OS: Windows 10 x64
- ggml: Vulkan backend (`--device vulkan`), weights as described per row
  (AR fp16 = `gsv-ar-f16.gguf`, AR q8 = `gsv-ar-q8.gguf`; cond/DiT F16; refcode F16)
- vocoder: ggml HiFi-GAN, F32 weights (`gsv-vocoder-f32.gguf`), Vulkan backend
- torch: 2.12.1 + CUDA 13.0, fp16, `use_cuda_graph=True`, batch size 1, text split `cut0`
- sample steps: v5turbo 4 / cfg 0.0; v5dev 32 / cfg 1.30
- cache-dit: DBCache `Fn=8`, threshold 0.02, warmup 8, downsample 4

Notes:

- Text: `你好，欢迎使用语音合成系统。`
- Reference: 16 kHz Chinese prompt, ~3.4 s output audio in every row.
- Timings vary by a few percent between runs; the torch numbers include CUDA graph replay
  (two warm-up runs with the same text are done before the measured run).

## DiT fusion (RoPE + adaLN)

`gsv_dit.cpp` builds the DiT step graph with a fused RoPE (`ggml_rope_ext`, head 0 only, with a
`pos` tensor and freq factors that reproduce the checkpoint `inv_freq`) and a fused
`LN(x)·(1+scale)+shift` (`ggml_layernorm_affine` with `wb = [1+scale, shift]`), replacing the
per-block cos/sin multiply chain and the `norm → add → mul → add` sequence.

Same machine, turbo-q8, same text / seed, warm:

| DiT | before | after | delta |
|---|---|---|---|
| v5turbo DiT | 333 ms | 291 ms | −12.6% |
| v5turbo total | 1130 ms | 1033 ms | −8.6% |

Audio correlation against the pre-fusion build is 0.9955; two runs with the same seed are
bit-identical (corr 1.000000), so determinism is preserved. The `mish` epilogue fusion (A4 in the
DiT report) is also in: `ggml_add_act` gained an `act=3` (mish) branch on CPU and in the Vulkan
shader, and `conv_pos` folds bias + mish into one node. Its effect is below the 0.2% noise floor,
so the numbers above do not change. `tests/test_add_act.cpp` checks all four activations against
a host reference on CPU and on every GPU backend; CI runs it under Mesa lavapipe, which exercises
the Vulkan shader branch (max|diff| 2.4e-7).

## CPU backend threading

ggml's barrier spins while waiting, so on SMT CPUs the default hardware_concurrency() (logical
threads) oversubscribes and slows requests down. gsv_server now defaults to physical cores when
--threads is not given (hardware_concurrency()/2 when > 8).

Windows, Xeon E5-2675 v3 (16 cores / 32 threads, 1.8 GHz), turbo-q8, 4 steps, 3.04 s audio:

| threads | total | DiT | cond | AR | vocoder |
|---|---|---|---|---|---|
| 32 (old default) | 39.9 s | 28.8 s | 1.80 s | 2.26 s | 3.07 s |
| 16 (new default) | 35.4 s | 29.0 s | 1.20 s | 1.52 s | 1.62 s |
| 8 | 35.9 s | 28.8 s | 1.15 s | 1.54 s | 1.93 s |

DiT is memory-bound in this configuration and does not improve past 8 threads; the win is in the
AR/cond/vocoder stages. --threads N overrides the default.

## What is not parallelised

- **Pipeline overlap** (AR of segment i+1 while DiT/vocoder runs segment i) is not implemented:
  ggml exposes a single backend queue, so both stages would serialize on the device anyway, and
  on CPU they compete for the same cores. Batches are formed *across* requests instead
  (/batch, AR batcher, request scheduler).
- **Larger conv tiles** in conv_direct_1d were not beneficial: the kernel already uses 6-wide
  time blocks x 16-wide output blocks, and DiT (the dominant CPU stage) is memory-bound, so
  thread count beyond 8 did not move it.