anira-project/moonshine-streaming

Moonshine Streaming for anira

Moonshine AI's streaming speech-to-text models, tiny and small, English and German, with every tensor at a fixed size, for real-time inference with anira on ONNX Runtime, ExecuTorch (CPU and Core ML) and Core ML. The fast engine of anira foundation models (speech to text): words appear while someone speaks.

Folder	Model	Checkpoint
tiny-en/	Moonshine Streaming Tiny, English	moonshine-ai/moonshine-streaming-tiny @ f8e9dfd
small-en/	Moonshine Streaming Small, English	moonshine-ai/moonshine-streaming-small @ 2c03650
tiny-de/	Moonshine Streaming Tiny, German	moonshine-ai/moonshine-streaming-tiny-de @ 928bfb9
small-de/	Moonshine Streaming Small, German	moonshine-ai/moonshine-streaming-small-de @ 56473ba

Fixed sizes

Moonshine's own runtime grows its tensors with every call: the encoder's features, the decoder's memory and its self-attention cache. Here the model is two graphs whose tensors never change shape, built from the checkpoints above in transformers (export/moonshine.py), with the limits Moonshine's runtime works within (moonshine-ai/moonshine, core/):

• a segment is at most 15 s plus its voice activity detector's 0.512 s look-behind: the decoder's memory holds 800 encoder frames (16 s at 50 Hz), masked beyond the segment's length;
• decoding emits at most 6.5 tokens per second of audio: the self-attention cache holds 128 positions, masked beyond the current one;
• the encoder's sliding windows (16 frames behind and 4 ahead per layer, read as Moonshine's runtime reads them) need 16 frames of context per layer and 16 frames of lookahead: the encoder runs on that window around the 25 new frames of each 0.5 s step (Moonshine's update interval), masked before the segment's start and after the audio so far.

Graph	Inputs	Outputs
encoder	audio [1,8000], conv1_buffer [1,D,4], conv2_buffer [1,2D,4], features [1,C+16,D], position [1], length [1]	k_cross [L,1,H,25,K], v_cross [L,1,H,25,K], conv1_buffer_out, conv2_buffer_out, features_out
decoder	token [1,1], position [1], k_self [L,1,H,128,K], v_self [L,1,H,128,K], k_cross [L,1,H,800,K], v_cross [L,1,H,800,K], memory_length [1]	logits [1,V], k_new [L,1,H,1,K], v_new [L,1,H,1,K]

The encoder is the frontend, the encoder, the adapter and the decoder's cross-attention projections in one step: 0.5 s of audio and the frontend's and encoder's state in, the cross-attention keys and values of the 25 frames that became final out (frames position to position + 24; frames outside the segment's length so far are not final). At the end of a segment one more step with silence and the same length flushes the last 16 frames. The decoder is one greedy step: the token, its position, the cache and the segment's memory in, the logits and the token's keys and values (to store at position) out. token, position, length and memory_length are int32.

Model	D	C (context)	L (decoder layers)	H	K	V
tiny-en	320	96	6	8	40	32768
small-en	620	160	10	8	64	32768
tiny-de	320	96	6	8	40	12288
small-de	620	160	10	8	64	12288

Files

Per folder: encoder.onnx, decoder.onnx; executorch/{encoder,decoder}.pte (CPU, XNNPACK) and executorch/{encoder,decoder}_coreml.pte (Core ML, Apple); coreml/{encoder,decoder}.mlpackage; and tokenizer.bin, Moonshine's tokenizer. All float32. Sizes and SHA256 of every file are in anira foundation models' catalog.

Format	Files	Encoder per 0.5 s step, decoder per token (M-series Mac; tiny / small)	Max deviation from PyTorch (encoder, decoder)
ONNX Runtime (CPU)	encoder.onnx, decoder.onnx	6 / 35 to 36 ms, 1.9 to 2.0 / 4.6 to 5.0 ms	1.6e-5, 1.9e-6
ExecuTorch, CPU (XNNPACK)	executorch/*.pte	11 to 12 / 44 to 49 ms, 4.1 to 4.3 / 9.8 ms	1.3e-5, 1.9e-6
ExecuTorch, Core ML	executorch/*_coreml.pte	4 to 5 / 7 to 10 ms, 2.4 to 4.0 / 4.3 to 5.1 ms	1.0e-5, 1.3e-6
Core ML ML program (macOS 13, iOS 16)	coreml/*.mlpackage	3.5 to 4 / 10 to 11 ms, 4.5 to 5.2 / 5.0 to 5.3 ms	1.0e-5, 1.3e-6

Deviations are relative to each output's peak, over every call of the test speech streamed in 0.5 s steps with each export's own state fed back and decoded after every step (as the live transcript does). Every export transcribes the test speech exactly as transformers does on the whole segment. The CPU programs also run on the ExecuTorch runtime anira ships (anira-project/backends v2.4.0). Moonshine's own int8 graphs (upstream/) transcribe the test speech as transformers does but for one word on tiny-en.

The graphs are written for XNNPACK, which has no layer norm and no 4-D batched matmul: LayerNorm's means are matrix products, attention runs as 3-D batched matmuls on slices of the caches, rotary embedding is a matrix product and the frontend's asinh is written out (Core ML has none); numerically the same.

cd export
uv run export_onnx.py <folder> <variant> <audio.wav>...
uv run export_executorch.py <folder> <variant> [--coreml] <audio.wav>...
uv run export_coreml.py <folder> <variant> <audio.wav>...
uv run check.py <variant> <audio.wav>...      # Moonshine's own graphs (upstream/) against transformers

upstream/ holds Moonshine's own runtime graphs (int8, growing tensors) unchanged, from moonshine-ai/moonshine-voice-assets @ 0bf2f2e (model/{tiny,small}-streaming-en/quantized_26_08_21/) and download.moonshine.ai (model/{tiny,small}-streaming-de/quantized_26_08_24/), for reference.

Use with anira

Pin a revision and check each file's SHA256 (in the catalog) when you download:

https://huggingface.co/anira-project/moonshine-streaming/resolve/<revision>/tiny-en/encoder.onnx

Licence

MIT. Moonshine's licence states that its models are released under the MIT License by default, in every language and at every size, including all streaming speech-to-text models; only legacy non-streaming models for languages other than English are under the Moonshine Community License, and none of those are here. LICENSE is Moonshine's licence file, copied from moonshine-ai/moonshine @ 234f60f. The export scripts (export/) are Apache-2.0.