AlexeyNe/supertonic-3-android-onnx

Supertonic 3 — ONNX for Android CPUs

Supertone/supertonic-3 (revision 3cadd1ee) re-laid for ONNX Runtime on phone CPUs. The voices and the outputs are the same as the original. The graphs are changed so that a low-end ARM CPU can run them in real time.

File	What changed
duration_predictor.onnx	unchanged
text_encoder.onnx	weights stored as fp16, cast to fp32 at load
vector_estimator.onnx	1×1 Conv → MatMul; dynamic int8 (per-channel) on those MatMuls only
vocoder.onnx	1×1 Conv → MatMul; weights stored as fp16, cast to fp32 at load
tts.json, unicode_indexer.bin, voice.bin	from the sherpa-onnx build

The reason for each change:

• 1×1 Conv → MatMul. ONNX Runtime's CPU Conv splits a pointwise convolution only across the batch, which is 2 for the vector estimator (it runs the conditional and unconditional passes together) and 1 for the vocoder. That leaves most cores idle. A MatMul is split across all of them. The rewrite is Transpose · MatMul · Transpose, and its outputs are bit-identical to the original.
• int8 only in the vector estimator's pointwise layers. The vector estimator runs once per diffusion step, so it carries most of the cost, and almost all of that cost is in the former 1×1 convolutions. Attention and the other MatMuls stay fp32. Quantizing them gains no speed and moves the sound away from the original. A dynamically quantized vocoder adds audible noise, so it stays fp32.
• fp16 storage. ONNX Runtime folds the Cast of an initializer when the session is created. The weights take half the space on disk and run at the same speed.

Measured on a Samsung Galaxy A12s (Exynos 850, 8× Cortex-A55, no dot-product), ONNX Runtime 1.24.2 with 8 intra-op threads:

Build	Real-time factor
sherpa-onnx int8, 2 threads, 8 steps	5.1
sherpa-onnx int8, 8 threads, 5 steps	3.0
this build, 8 threads, 5 steps	0.8

Use at least 5 diffusion steps. At 4 steps the output keeps residual noise in the high frequencies, most audibly in Russian.

The conversion is reproducible: uv run build-supertonic-pack.py OUT (script included here).

License: BigScience OpenRAIL-M, the same as the original model (see LICENSE). Its use restrictions apply to this build too.