Add an engine backend: the 22-block DiT runs on hand-written WebGPU
kernels (tiled/sgmat GEMM, fused LN+adaLN, RoPE, tiled QK/softmax/AV,
grouped convpos, GRN) with 4-bit packed weights and per-group scales
dequantized in-kernel. The worker loads only the ORT vocoder for this
path; ONNX Runtime remains the fallback.
- 229.7 MB int4 bundle, ~1.2 s per clip on WebGPU (was 2.8 s int4 ORT)
- flow: 4 steps, sway -1, cfg 0 (B=1, no null pass)
- tuning: tts.backend / tts.gemm switch kernels without touching the worker
- fixes found on the way: u32 vs f32 uniforms, second SiLU on the time MLP,
RoPE inv_freq exponent, AV score-tile column, f32 attention scores,
safe tanh in Mish
2 days ago
Serve voice refs from the Space origin; prefetch at init
The reference mel was fetched from the HF model repo during the first
synthesis: two redirects to the CDN, ~3 s for 112 KB. Ship the refs with
the Space (112 KB) and load them same-origin, prefetching during init.
Synthesis on the live site drops from ~7 s to ~3 s.
Switch the runtime to the reflow-distilled 4-step model:
- worker: cfgStrength from tuning/manifest, skip the null pass at cfg=0
- tuning: flowSteps 4, cfgStrength 0
- model repo: dit_step_4s_int4.onnx (Split->Slice + aux outputs so JSEP
does not miscompile it) and vocoder_4s_int4.onnx; manifest steps=4, cfg=0
- README: 435 MB download, few-seconds synthesis on WebGPU
Synthesis on this M3 Air: ~2.8-3.4 s per short clip on WebGPU (was ~22 s at
32 steps, ~8 s at 16), ~31 s on the wasm fallback.
Quantization is not a speed win on this pair of runtimes: 4-bit
MatMulNBits runs 3.7x slower on the WebGPU EP, 8-bit is unsupported
there, and int8/int4 wasm time is unchanged. The DiT is compute-bound,
so cutting sampler steps is the lever; 16 steps gives 8.2s vs 22.0s
with identical ASR transcripts (same-seed mel cosine 0.984).
- tts.worker.js: rewrite synthesis to the F5-TTS zero-shot layout
(cond = ref mel + zero tail, x = whole-length noise, text_ids =
ref+gen, slice gen frames after the flow) and RMS-normalize the
PCM to 0.1; fixes unintelligible output on both backends.
- New gemma-style UI (index.html/app.js), self-hosted fonts,
coi-serviceworker, trimmed ort runtime (ort.mjs + jsep pair).
- Load fp16 dit/vocoder graphs on WebGPU (int8 wasm fallback).
- README: accurate download sizes and fallback note.
