Matcha-TTS + HiFi-GAN v2 for Sentis (com.sky.sentis.matcha.v2)

Unity Sentis (com.unity.ai.inference 2.6.1) port of Matcha-TTS (OT-CFM flow matching, LJSpeech English) with the small HiFi-GAN v2 vocoder (3.6 MB instead of 53 MB). It runs fp32 and unquantised. The acoustic model is exported from the author's PyTorch checkpoint (matcha_ljspeech.ckpt); the v2 vocoder is the sherpa-onnx HiFi-GAN v2 graph, wrapped to the same mel_norm → wav contract as v1. The Euler ODE loop is driven from C#, so the step count is a runtime knob.

For the larger original HiFi-GAN v1 vocoder (slightly higher naturalness), see com.sky.sentis.matcha.v1.

using MatchaV2;
using Unity.InferenceEngine;

using var tts = new MatchaTts(backend: BackendType.GPUCompute);   // loads StreamingAssets/MatchaV2
tts.Steps = 5;                                                      // 2 = fastest, 5 = default
tts.Warmup();

var phrases = MatchaPhrases.Load(Path.Combine(MatchaTts.DefaultModelDirectory, "phrases_en.json"));
float[] pcm = tts.Synthesize(phrases[0].Sentences);                // 22.05 kHz mono
var clip = AudioClip.Create("tts", pcm.Length, 1, MatchaTts.SampleRate, false);
clip.SetData(pcm, 0);

Install

Add the package by git URL, then run Tools ▸ Matcha-TTS v2 ▸ Install Models to StreamingAssets (or call MatchaV2.Editor.MatchaBaker.Bake()). Model~ is hidden from the asset pipeline and not included in player builds, so the .sentis files are copied to Assets/StreamingAssets/MatchaV2.

https://huggingface.co/Sky-Kim/com.sky.sentis.matcha.v2

Text frontend (offline)

The graphs take phoneme ids: english_cleaners2 → espeak-ng en-us IPA (with stress and punctuation) → Matcha symbol table → a blank (id 0) interspersed between symbols. There is no in-Unity G2P. Model~/phrases_en.json holds precomputed ids for a test corpus; produce ids for your own text with Tools~/make_fixture.py (see Tools~/common.py: text_to_ids).

Layout

Path	What
Model~/matcha_encoder.sentis	text encoder + duration predictor: x[1,T] → mu_x[1,80,T], logw[1,1,T]
Model~/matcha_estimator.sentis	one Euler step of the CFM decoder: x, mask, mu [1,80,L], t[1] → v[1,80,L]
Model~/hifigan_v2.sentis	HiFi-GAN v2: mel_norm[1,80,F] → wav[1,F·256] (denormalisation and clamp inside)
Model~/phrases_en.json	precomputed ids for the benchmark corpus, split into sentences
Model~/fixtures/	ONNX Runtime / PyTorch reference tensors used by the tests
Runtime/	MatchaTts (runner), MatchaPhrases
Editor/MatchaBaker.cs	copies Model~ into StreamingAssets
Tests/Editor/	Sentis-vs-ORT parity per graph (CPU, GPUCompute) and end-to-end synthesis
Tools~/	export (export.py), v2 wrapper (wrap_hifigan_v2.py), fixtures/phrases (make_fixture.py), ORT reference chain

Pipeline

1. Encoder on the ids.
2. Length regulation on the host (MatchaTts.Expand): w = ceil(exp(logw)) · length_scale, y_len = max(1, floor(Σw)), L = ceil(y_len/4)·4; frame j copies encoder column i where cum[i-1] ≤ j < cum[i].
3. x = N(0,1) · temperature (seeded), then for k = 0..N-1: v = estimator(x, mask, mu_y, k/N), x += v/N. The update runs in a compiled Functional graph, so x stays on the device.
4. x[:, :, :y_len] → HiFi-GAN v2.

Defaults: temperature 0.667, length_scale 1.0, N = 5. The optional HiFi-GAN denoiser of the Matcha CLI is not included.

Verification (Sentis 2.6.1, Unity 6000.5.5f1, Apple Silicon)

Edge	Result
PyTorch (original) vs ONNX Runtime	mu_x 2.3e-6, logw 4.6e-6, mel 4.9e-6 (N=5); wrapped v2 vs raw sherpa-onnx graph 0.0
ORT vs Sentis CPU	encoder ≤ 4.3e-6, estimator 5.1e-6, vocoder 4.5e-6 (waveform SNR 109 dB)
ORT vs Sentis GPUCompute	encoder ≤ 4.3e-6, estimator 5.7e-6, vocoder 2.7e-5 (waveform SNR 99 dB)
Back-transcription (Whisper small)	WER 6.0 % over the 5-phrase corpus; UTMOS 3.43

Benchmark (Apple M5 Max, standalone player, warm median)

	RTF (↓)	short-phrase TTFA
GPUCompute, 5 steps	0.006	35 ms
CPU, 5 steps	0.022	82 ms

About 2.5× faster than HiFi-GAN v1 (GPU RTF 0.015) for a small drop in UTMOS (3.67 → 3.43). Measure on your own target device.

License

Matcha-TTS code and LJSpeech checkpoint © Shivam Mehta et al., MIT. HiFi-GAN © Jungil Kong (jik876), MIT; the v2 ONNX graph comes from k2-fsa/sherpa-onnx (Apache-2.0). LJSpeech is public domain. See LICENSE.md.