SenseVoice-Small fixed-shape LiteRT buckets (3/5/10/30 s), w8 + i8

17eeff4 verified 6 days ago

6.78 kB

---
license: other
license_name: funasr-model-license
license_link: https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE
base_model:
- FunAudioLLM/SenseVoiceSmall
language:
- ko
- en
- zh
- ja
- yue
tags:
- litert
- tflite
- sensevoice
- asr
- quantized
- korean
- xnnpack
pipeline_tag: automatic-speech-recognition
---
# SenseVoice-Small — fixed-shape LiteRT (TFLite) buckets, int8

[FunAudioLLM/SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)
(~234 M params, non-autoregressive SAN-M encoder + CTC) converted to LiteRT for
on-device use, unchanged weights. One forward pass per utterance — no decoder loop.

## What is different from the upstream ONNX

- **Static shapes, one graph per window** (3 / 5 / 10 / 30 s). A dynamic-shape graph keeps
  the XNNPACK delegate from attaching on LiteRT; these graphs run 10× (fp32) to 25× (i8)
  faster with XNNPACK than on the built-in kernels.
- **Padding is masked, not heard.** `speech_lengths` stays a graph input: pad the features
  of a shorter clip with zeros up to the bucket and feed the true length. Measured: padded +
  true length is bit-identical to the unpadded graph (max |Δlogit| 0.0000). Feeding padded
  *audio* without the length instead wrecks it (real-command CER 0.009 → 0.44 at a 5 s pad).
- **Weights int8.** `w8` = `weight_only_wi8_afp32` (activations fp32), `i8` =
  `dynamic_wi8_afp32` (activations quantized per call — faster, slightly worse in noise).

## Files

| File | Size | Window | `speech` input | Note |
| --- | ---: | --- | --- | --- |
| `sensevoice_small_3s_w8.tflite` / `_i8` | 231 MB | 3 s | `[1, 560, 50]` | short commands |
| `sensevoice_small_5s_w8.tflite` / `_i8` | 231 MB | 5 s | `[1, 560, 84]` | **voice commands, recommended** |
| `sensevoice_small_10s_w8.tflite` / `_i8` | 231 MB | 10 s | `[1, 560, 167]` | sentences |
| `sensevoice_small_30s_w8.tflite` / `_i8` | 231 MB | 30 s | `[1, 560, 500]` | dictation |
| `am.mvn` | 11 KB | | | CMVN (AddShift + Rescale, 560 each) |
| `chn_jpn_yue_eng_ko_spectok.bpe.model` | 0.4 MB | | | SentencePiece, 25055 pieces |
| `tokens.txt` | 0.3 MB | | | `id<TAB>piece`, same vocab, for runtimes without SentencePiece |
| `config.yaml` | | | | upstream config (front-end parameters) |
| `SHA256SUMS` | | | | |

Load only the bucket(s) you need — every graph carries the full weights.

## REQUIRED integration notes (tensor contract)

Bind inputs **by name** (signature `serving_default`; the converter orders them alphabetically).

| Tensor | Shape | Type | Value |
| --- | --- | --- | --- |
| `speech` | `[1, 560, N]` | f32 | LFR+CMVN features, **feature-major** (transpose of the usual `[T, 560]`), zero-padded to N |
| `speech_lengths` | `[1]` | i32 | true number of LFR frames (≤ N) |
| `language` | `[1]` | i32 | query id: auto 0, zh 3, en 4, yue 7, ja 11, **ko 12**, nospeech 13 |
| `textnorm` | `[1]` | i32 | `withitn` 14 (punctuation + digits), `woitn` 15 |
| `ctc_logits` (out) | `[1, N+4, 25055]` | f32 | keep the first `speech_lengths + 4` rows |
| `encoder_out_lens` (out) | `[1]` | i32 | = `speech_lengths + 4` |

Front end (must match exactly — this is funasr `WavFrontend`):

1. 16 kHz mono, waveform × 32768.
2. Kaldi fbank (`torchaudio.compliance.kaldi.fbank`): 80 mel bins, 25 ms window / 10 ms
   shift, Hamming, `snip_edges=True`, `energy_floor=0`, **dither 0** (funasr defaults to 1.0,
   which makes runs non-deterministic); Kaldi defaults otherwise (pre-emphasis 0.97, DC removal).
3. LFR m=7, n=6: prepend 3 copies of the first frame, stack 7 frames every 6, pad the tail
   with the last frame (→ 560-dim, 60 ms per frame, `ceil(T/6)` frames).
4. CMVN from `am.mvn`: `(x + shift) * scale`.

Decode: greedy CTC — argmax per row, collapse repeats, drop id 0 (blank), map ids through
`tokens.txt` (`▁` = space). The first four emitted tokens are tags
(`<|ko|>` `<|NEUTRAL|>` `<|Speech|>` `<|woitn|>`); strip `<|…|>`.

## ⚠️ Evaluation caveats — read before using the numbers

- `gate40` is the 40-clip **TTS-synthesized** Korean short-command holdout used to rank this
  project's Whisper ACFT tiers. It is valid for **ranking only** (Spearman ρ = 1.00 against
  real recordings) and runs ~2.8× harsher than real audio in absolute CER.
- References average 3.6 characters: one wrong syllable is CER 0.25–0.33.
- Zeroth-Korean test sentences also appear in Zeroth train (456 / 457), and SenseVoice's
  training data is undisclosed — treat the Zeroth number as optimistic.
- CER = Levenshtein on the string with spaces removed and punctuation stripped (what a
  command matcher sees), `woitn`, language forced to `ko`.

## Validation (desktop, Windows x86-64 CPU, LiteRT 2.1.6, XNNPACK, 4 threads)

| Set | n | PyTorch (funasr, CUDA) | tflite fp32 | **w8** | i8 |
| --- | ---: | ---: | ---: | ---: | ---: |
| gate40 (TTS, ranking only) | 40 | 0.169 · 21 exact | 0.204 · 20 | 0.204 · 20 | 0.204 · 20 |
| real commands, quiet room | 13 | 0.009 · 11 | 0.006 · 12 | **0.006 · 12** | 0.006 · 12 |
| real commands, field noise | 9 | 0.296 · 4 | 0.269 · 5 | **0.269 · 5** | 0.333 · 5 |
| Zeroth-Korean test (see caveat) | 457 | 0.024 | 0.023 | 0.023 | 0.024 |
| FLEURS ko dev | 64 | 0.071 | 0.068 | 0.068 | 0.068 |

- **Conversion parity**: fp32 tflite vs onnxruntime on the same features — max |Δlogit|
  ≤ 3.6e-4 on commands, 0.029 on 20 s sentences, CTC argmax agreement 100 %, transcripts
  identical on all 583 clips. The PyTorch column differs only by CUDA-vs-CPU numerics
  flipping borderline tokens (e.g. `확대` → kana in both, on different clips).
- **Against Whisper ACFT-KO** on the same gate, same protocol (first 24 clips, punctuation
  kept): SenseVoice-Small 0.198 (PyTorch) / 0.247 (tflite) vs turbo-acft 0.182, medium
  0.208, base 0.305, tiny 0.457 — between turbo and base, at a quarter of turbo's size.

Invoke latency per window (model only, p50):

| Window | fp32 | w8 | i8 |
| --- | ---: | ---: | ---: |
| 3 s | 128 ms | 134 ms | 98 ms |
| 5 s | 145 ms | 172 ms | 121 ms |
| 10 s | 290 ms | 286 ms | 211 ms |
| 30 s | 827 ms | 815 ms | 691 ms |

Not yet measured on an Android device.

## Caveats

- Stock weights. Two Korean fine-tunes were measured and lost to stock on every set
  (an EDIE fine-tune: Zeroth CER 0.48; an in-house Zeroth fine-tune: real-command CER 0.020 vs 0.009).
- Field noise is the weak spot (fan/wind/vehicle recordings: 5 / 9 exact).
- Language forcing is soft: a few 1-syllable clips come out in kana even with `ko`.

## License / attribution

Weights: FunAudioLLM SenseVoiceSmall, under the
[FunASR model license](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE).
Conversion only — no weights were changed. Paper: [FunAudioLLM, arXiv 2407.04051](https://arxiv.org/abs/2407.04051).

	---
	license: other
	license_name: funasr-model-license
	license_link: https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE
	base_model:
	- FunAudioLLM/SenseVoiceSmall
	language:
	- ko
	- en
	- zh
	- ja
	- yue
	tags:
	- litert
	- tflite
	- sensevoice
	- asr
	- quantized
	- korean
	- xnnpack
	pipeline_tag: automatic-speech-recognition
	---

	# SenseVoice-Small — fixed-shape LiteRT (TFLite) buckets, int8

	[FunAudioLLM/SenseVoiceSmall](https://huggingface.co/FunAudioLLM/SenseVoiceSmall)
	(~234 M params, non-autoregressive SAN-M encoder + CTC) converted to LiteRT for
	on-device use, unchanged weights. One forward pass per utterance — no decoder loop.

	## What is different from the upstream ONNX

	- Static shapes, one graph per window (3 / 5 / 10 / 30 s). A dynamic-shape graph keeps
	the XNNPACK delegate from attaching on LiteRT; these graphs run 10× (fp32) to 25× (i8)
	faster with XNNPACK than on the built-in kernels.
	- Padding is masked, not heard. `speech_lengths` stays a graph input: pad the features
	of a shorter clip with zeros up to the bucket and feed the true length. Measured: padded +
	true length is bit-identical to the unpadded graph (max \|Δlogit\| 0.0000). Feeding padded
	audio without the length instead wrecks it (real-command CER 0.009 → 0.44 at a 5 s pad).
	- Weights int8. `w8` = `weight_only_wi8_afp32` (activations fp32), `i8` =
	`dynamic_wi8_afp32` (activations quantized per call — faster, slightly worse in noise).

	## Files

	\| File \| Size \| Window \| `speech` input \| Note \|
	\| --- \| ---: \| --- \| --- \| --- \|
	\| `sensevoice_small_3s_w8.tflite` / `_i8` \| 231 MB \| 3 s \| `[1, 560, 50]` \| short commands \|
	\| `sensevoice_small_5s_w8.tflite` / `_i8` \| 231 MB \| 5 s \| `[1, 560, 84]` \| voice commands, recommended \|
	\| `sensevoice_small_10s_w8.tflite` / `_i8` \| 231 MB \| 10 s \| `[1, 560, 167]` \| sentences \|
	\| `sensevoice_small_30s_w8.tflite` / `_i8` \| 231 MB \| 30 s \| `[1, 560, 500]` \| dictation \|
	\| `am.mvn` \| 11 KB \| \| \| CMVN (AddShift + Rescale, 560 each) \|
	\| `chn_jpn_yue_eng_ko_spectok.bpe.model` \| 0.4 MB \| \| \| SentencePiece, 25055 pieces \|
	\| `tokens.txt` \| 0.3 MB \| \| \| `id<TAB>piece`, same vocab, for runtimes without SentencePiece \|
	\| `config.yaml` \| \| \| \| upstream config (front-end parameters) \|
	\| `SHA256SUMS` \| \| \| \| \|

	Load only the bucket(s) you need — every graph carries the full weights.

	## REQUIRED integration notes (tensor contract)

	Bind inputs by name (signature `serving_default`; the converter orders them alphabetically).

	\| Tensor \| Shape \| Type \| Value \|
	\| --- \| --- \| --- \| --- \|
	\| `speech` \| `[1, 560, N]` \| f32 \| LFR+CMVN features, feature-major (transpose of the usual `[T, 560]`), zero-padded to N \|
	\| `speech_lengths` \| `[1]` \| i32 \| true number of LFR frames (≤ N) \|
	\| `language` \| `[1]` \| i32 \| query id: auto 0, zh 3, en 4, yue 7, ja 11, ko 12, nospeech 13 \|
	\| `textnorm` \| `[1]` \| i32 \| `withitn` 14 (punctuation + digits), `woitn` 15 \|
	\| `ctc_logits` (out) \| `[1, N+4, 25055]` \| f32 \| keep the first `speech_lengths + 4` rows \|
	\| `encoder_out_lens` (out) \| `[1]` \| i32 \| = `speech_lengths + 4` \|

	Front end (must match exactly — this is funasr `WavFrontend`):

	1. 16 kHz mono, waveform × 32768.
	2. Kaldi fbank (`torchaudio.compliance.kaldi.fbank`): 80 mel bins, 25 ms window / 10 ms
	shift, Hamming, `snip_edges=True`, `energy_floor=0`, dither 0 (funasr defaults to 1.0,
	which makes runs non-deterministic); Kaldi defaults otherwise (pre-emphasis 0.97, DC removal).
	3. LFR m=7, n=6: prepend 3 copies of the first frame, stack 7 frames every 6, pad the tail
	with the last frame (→ 560-dim, 60 ms per frame, `ceil(T/6)` frames).
	4. CMVN from `am.mvn`: `(x + shift) * scale`.

	Decode: greedy CTC — argmax per row, collapse repeats, drop id 0 (blank), map ids through
	`tokens.txt` (`▁` = space). The first four emitted tokens are tags
	(`<\|ko\|>` `<\|NEUTRAL\|>` `<\|Speech\|>` `<\|woitn\|>`); strip `<\|…\|>`.

	## ⚠️ Evaluation caveats — read before using the numbers

	- `gate40` is the 40-clip TTS-synthesized Korean short-command holdout used to rank this
	project's Whisper ACFT tiers. It is valid for ranking only (Spearman ρ = 1.00 against
	real recordings) and runs ~2.8× harsher than real audio in absolute CER.
	- References average 3.6 characters: one wrong syllable is CER 0.25–0.33.
	- Zeroth-Korean test sentences also appear in Zeroth train (456 / 457), and SenseVoice's
	training data is undisclosed — treat the Zeroth number as optimistic.
	- CER = Levenshtein on the string with spaces removed and punctuation stripped (what a
	command matcher sees), `woitn`, language forced to `ko`.

	## Validation (desktop, Windows x86-64 CPU, LiteRT 2.1.6, XNNPACK, 4 threads)

	\| Set \| n \| PyTorch (funasr, CUDA) \| tflite fp32 \| w8 \| i8 \|
	\| --- \| ---: \| ---: \| ---: \| ---: \| ---: \|
	\| gate40 (TTS, ranking only) \| 40 \| 0.169 · 21 exact \| 0.204 · 20 \| 0.204 · 20 \| 0.204 · 20 \|
	\| real commands, quiet room \| 13 \| 0.009 · 11 \| 0.006 · 12 \| 0.006 · 12 \| 0.006 · 12 \|
	\| real commands, field noise \| 9 \| 0.296 · 4 \| 0.269 · 5 \| 0.269 · 5 \| 0.333 · 5 \|
	\| Zeroth-Korean test (see caveat) \| 457 \| 0.024 \| 0.023 \| 0.023 \| 0.024 \|
	\| FLEURS ko dev \| 64 \| 0.071 \| 0.068 \| 0.068 \| 0.068 \|

	- Conversion parity: fp32 tflite vs onnxruntime on the same features — max \|Δlogit\|
	≤ 3.6e-4 on commands, 0.029 on 20 s sentences, CTC argmax agreement 100 %, transcripts
	identical on all 583 clips. The PyTorch column differs only by CUDA-vs-CPU numerics
	flipping borderline tokens (e.g. `확대` → kana in both, on different clips).
	- Against Whisper ACFT-KO on the same gate, same protocol (first 24 clips, punctuation
	kept): SenseVoice-Small 0.198 (PyTorch) / 0.247 (tflite) vs turbo-acft 0.182, medium
	0.208, base 0.305, tiny 0.457 — between turbo and base, at a quarter of turbo's size.

	Invoke latency per window (model only, p50):

	\| Window \| fp32 \| w8 \| i8 \|
	\| --- \| ---: \| ---: \| ---: \|
	\| 3 s \| 128 ms \| 134 ms \| 98 ms \|
	\| 5 s \| 145 ms \| 172 ms \| 121 ms \|
	\| 10 s \| 290 ms \| 286 ms \| 211 ms \|
	\| 30 s \| 827 ms \| 815 ms \| 691 ms \|

	Not yet measured on an Android device.

	## Caveats

	- Stock weights. Two Korean fine-tunes were measured and lost to stock on every set
	(an EDIE fine-tune: Zeroth CER 0.48; an in-house Zeroth fine-tune: real-command CER 0.020 vs 0.009).
	- Field noise is the weak spot (fan/wind/vehicle recordings: 5 / 9 exact).
	- Language forcing is soft: a few 1-syllable clips come out in kana even with `ko`.

	## License / attribution

	Weights: FunAudioLLM SenseVoiceSmall, under the
	[FunASR model license](https://github.com/modelscope/FunASR/blob/main/MODEL_LICENSE).
	Conversion only — no weights were changed. Paper: [FunAudioLLM, arXiv 2407.04051](https://arxiv.org/abs/2407.04051).