s0undy/pianissimo-sv-ort

Pianissimo (Swedish) – ONNX Runtime format for ARM

Klang Pianissimo, the Swedish 600M-parameter FastConformer-TDT model by Klang AI AB, in ONNX Runtime's own format (.ort), with every graph optimization already applied for ARM CPUs. These are the files the Sona Android app downloads: they load with graph optimization off and can be memory-mapped, so the model is ready in seconds instead of minutes on a budget phone. The numbers they compute are the same as the ONNX models they come from.

Files	Sona's name	From
encoder-model.klang-int8.ort + decoder_joint-model.q4.ort	Standard (INT8)	The encoder of Klang's int8 build, KlangAI/pianissimo-sv-onnx at 63730c6
encoder-model.klang-int4.ort + decoder_joint-model.q4.ort	Kompakt (4-bit)	The encoder of Klang's int4 build, same repo and commit
decoder_joint-model.q4.ort	The decoder of both	moonhouse's INT8 decoder with its weights stored as uint8, from s0undy/pianissimo-sv-onnx-q4 at 10de7cc
vocab.txt, config.json	Tokens and the settings onnx-asr needs

Klang's own decoders are here too (decoder_joint-model.klang-int8.ort, decoder_joint-model.klang-int4.ort), but Sona does not use them: on a Galaxy A32 they took 7.7 ms and 41 ms per call against 4.7 ms for this decoder. The encoders and the decoder come from the same checkpoint.

Earlier versions of Sona download the files at older commits of this repo, which stay here: encoder-model.int8.ort and decoder_joint-model.int8.ort from moonhouse/pianissimo-sv-onnx at 72c3826, and encoder-model.q4.ort and decoder_joint-model.q4.ort from s0undy/pianissimo-sv-onnx-q4 at 10de7cc.

Accuracy, FLEURS Swedish, full test split, with Sona's pipeline: 6.65 % WER for Standard and 6.54 % for Kompakt (6.43 % and 6.73 % for the earlier builds). Klang reports 6.51 % for the original model; see the source repos.

Use

ONNX Runtime 1.30 or later (Sona uses 1.30.0), on ARM64:

import onnxruntime as ort
so = ort.SessionOptions()
so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_DISABLE_ALL  # already optimized
so.add_session_config_entry("session.use_memory_mapped_ort_model", "1")
so.add_session_config_entry("session.use_ort_model_bytes_for_initializers", "1")
encoder = ort.InferenceSession("encoder-model.klang-int4.ort", so)

The graphs are optimized for ARM (no x86-specific NCHWc layout). They also run on x86, but the results then differ slightly from the ONNX model on the same machine; convert the ONNX model yourself with --target_platform amd64 for x86.

Most of what remains of the load time is ONNX Runtime planning buffer reuse, which grows with the square of the node count. Klang's encoders have about 3,600 nodes after optimization, so it takes about a second on a desktop CPU; the earlier builds have 24,500 and took about 30 s. ONNX Runtime skips it in parallel execution mode with two inter-op threads (ExecutionMode.ORT_PARALLEL); with a single CPU stream the nodes still run one after another and the output stays bit-identical, but inference can be slower. Sona measures both on the phone and keeps the faster one.

How it was made

With research/model-benchmark/convert_ort.py, which runs ONNX Runtime 1.30.0's convert_onnx_models_to_ort with --optimization_style Fixed --target_platform arm on the source files as published: Klang's encoders and the decoder from s0undy/pianissimo-sv-onnx-q4. ONNX Runtime writes some tables in hash-map order, so a new conversion is not byte-identical; ort_digest.py hashes the graph and weights in a canonical order, and Sona's CI checks that a fresh conversion of the source models gives the same digest as these files.

Licence and attribution

CC BY 4.0, like the models they are made from:

• Klang Pianissimo by Klang AI AB, CC BY 4.0. Fine-tuned from NVIDIA Parakeet TDT 0.6B v3.
• The INT8 and int4 encoders are from Klang's own ONNX builds, CC BY 4.0.
• The decoder and the earlier INT8 files come from the ONNX conversion by moonhouse, CC BY 4.0; the earlier 4-bit files from Sona's own 4-bit build.
• ORT-format conversion for Sona.