TinyMOSS-Diarize

MOSS-Transcribe-Diarize 0.9B, quantized to an average of 2.911 bits per value — 1.8 GB down to 316 MiB — and still doing all three jobs in a single pass: transcription, timestamps, and speaker labels, in nine languages.

The weights here are stored packed. With the Arm NEON kernels in the companion repo they are multiplied as packed data, so a 4 GB Raspberry Pi can run a model it otherwise cannot load.

Quick start

pip install torch transformers soundfile
python run_inference.py your_audio.wav --device cpu

Output looks like this — timestamps and speaker tags come from the model itself, not a separate diarization pass:

[0.48][S01] He hoped there would be stew for dinner, turnips and carrots and bruised
potatoes and fat mutton pieces to be ladled out in thick, peppered flour fat and sauce.[10.34]

SHA256SUMS covers every file in this repo.

How the weights are stored

Part of the model	Tensors	Storage	Bits/value
Language decoder projections	196	1.25-bit codes, group of 128, FP16 scale	1.375
Audio encoder projections	144	4-bit codes, group of 128, FP16 scale	4.125
Embedding (tied with output head)	1	4-bit codes, per-channel scale	4.015625
Norms, biases, conv frontend	538	BF16, untouched	16.0
Total	879	330,968,414 bytes	2.911

Unpacking is bit-identical to the reference implementation (max_abs = 0), and the loader verifies all 341 packed tensors by checksum before use.

Accuracy

16 datasets, 553 utterances, zero inference errors and zero malformed transcripts under the grammar-constrained decoder.

Multi-speaker audio is where this model is worth its size. Whisper transcribes only, so it is scored on text alone and still trails badly:

Dataset	This model	whisper-large-v3-turbo	whisper-small
Chinese mixtures	7.50% cpWER	33.87% CER (text only)	120.53% CER (text only)
AliMeeting	21.86% cpWER	42.92% CER (text only)	44.56% CER (text only)
English mixtures	20.12% cpWER	7.01% WER (text only)	8.34% WER (text only)
Noisy English mixtures	23.52% cpWER	15.06% WER (text only)	18.98% WER (text only)

Single speaker, quantization costs us real accuracy:

Dataset	This model	Same model in BF16
LibriSpeech test-clean	3.48% WER	0.92% WER
AISHELL test	2.71% CER	1.50% CER
Noisy LibriSpeech	3.17% WER	1.33% WER

FLEURS (30 utterances per language): en 9.64%, de 16.00%, es 7.10%, fr 18.42%, it 10.87%, ko 23.57%, ru 19.40% WER; ja 12.10%, zh 11.12% CER. Whisper-turbo is better than this model on every FLEURS language — the trade here is size and diarization, not raw monolingual accuracy.

On-device performance

Same model, same audio, same machine; the only difference is whether weights stay packed.

Apple M4 MacBook Air, 10.4 s clip:

	packed	dequantized to BF16
Total	18.2 s	24.1 s
Time to first token	9.7 s	19.7 s
Load	0.54 s	14.7 s
Peak memory	3,998 MiB	7,011 MiB
Decode	17.2 tok/s	33.5 tok/s

Raspberry Pi 5, 4 GB, 5.2 s clip: packed finishes in 142.7 s (load 8.9 s, peak 3,077 MiB). The same model dequantized to BF16 never finished — 177 s just to load, still on its first token when killed at 628 s. FP32 managed one token before timing out.

Every on-device transcript matched the GPU reference character for character. Benchmarks are 1 warmup

• 3 timed runs, median. Raw data and the kernels are in the GitHub repo.

Limitations

• "2.911 bits" is an average across the package, not a uniform 3-bit model.
• Meeting-room diarization is the hardest regime; AliMeeting timestamp error averages 6.22 s.
• The decoder guarantees a parseable transcript; it does not infer or repair speaker identity.
• Clean single-speaker accuracy is measurably worse than the BF16 original.

License and attribution

Apache-2.0. Built on OpenMOSS/MOSS-Transcribe-Diarize (Apache-2.0). Quantization, packed-weight runtime, and Arm kernels added on top.