README.md

---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- automatic-speech-recognition
base_model:
- jonatasgrosman/wav2vec2-large-xlsr-53-japanese
---
# wav2vec2 XLSR-53 Japanese — ExecuTorch (CTC speech recognition)

Ten seconds of Japanese audio in, a transcript out, in **one forward pass**. No
decoder loop, no beam, no KV cache, nothing to carry between calls — the encoder
is the model and the transcript falls out of an argmax per 20 ms frame.

That is the difference from the Whisper conversions on this shelf, which are
encoder-decoder: one encoder pass per window plus one decoder pass per generated
token. A CTC model trades that machinery — and the ability to punctuate — for a
much simpler thing to run on a device.

```
(waveform (1,160000) fp32 16 kHz mono, attention_mask (1,160000) int64)
    -> logits (1, 499, 2341)
```

| build | file | MB | corr vs fp32 eager | transcripts identical | Mac ms* |
|---|---|---|---|---|---|
| fp32 | `wav2vec2_xlsr53_ja_xnnpack_fp32.pte` | 1271.7 | 1.000000 | **5/5** | 267.8 |
| int8 (dynamic) | `wav2vec2_xlsr53_ja_xnnpack_int8.pte` | **357.9** | 0.997838 | **5/5** | 228.9 |

\*Mac arm64, single process, median of 5, for a 10-second window — a reference
point for relative cost, not a device number. Torch eager fp32 on the same
machine: 215.1 ms. XNNPACK delegate coverage 64.5% (869/1347 ops) for fp32,
75.2% for int8.

**Take the int8 build.** It is 28% of the file, it is faster, and it returns the
same five transcripts character for character, at worst character error rate
0.000. Its correlation is 0.9978 rather than 1.000000, and on this task that
difference did not reach the output — but five sentences of synthetic speech is
what stands behind that, not a benchmark.

**There is no Core ML build.** coremltools rejects the graph on a `stack` whose
operands are a mix of int and float. This one is XNNPACK only, and the same file
runs on Android and on the Mac.

## Running it

**1. The audio.** 16 kHz mono in [-1, 1], exactly 160 000 samples. Zero-pad a
shorter clip; cut a longer one. **Do not normalise it** — the per-utterance
normalisation wav2vec2 expects is inside the graph, and it needs the mask to be
correct (below).

**2. The mask.** `attention_mask` is int64 `(1, 160000)`, 1 on the real samples
and 0 on the padding. It is not optional and it is not cosmetic: padding a
3.7-second clip to 10 seconds without it changed the transcript on **4 of 5** test
sentences.

```
with the mask:     今日はよい天気ですね
without:           今わよい電気で住
```

The zeros drag the mean and shrink the variance the normalisation divides by, so
the speech comes out at the wrong scale, and the encoder attends to the silence.
With the mask — masked mean and variance, and the same mask handed to the encoder
— all five come back identical to the transcript of the unpadded clip.

**3. The decode.** CTC greedy, over `logits[0]`:

```
ids = argmax per frame
collapse runs of the same id
drop the blank (id 0)
map through the repo's vocab.json, turn the word delimiter | into a space, strip
```

That last clause matters. Without the strip this recipe returns
`'今日はよい天気ですね '` where the model's own tokenizer returns
`'今日はよい天気ですね'` — a `|` lands at the end of the utterance. The checker
in the conversion repo runs the hand-written recipe and asserts it against
`processor.batch_decode` on the same ids, which is how that was caught.

There is no punctuation and no casing in the output: the vocabulary has 2341
entries and none of them are `。` or `、`. The model also writes numbers as
digits (`コーヒーを1杯`).

## Verification

Five sentences spoken by macOS `say -v Kyoko`, transcribed through the `.pte` and
through the eager model on the identical padded window and mask:

```
今日はよい天気ですね
駅までの道を教えてください
この電車は東京駅に泊まりますか
コーヒーを1杯お願いします
厚の会議は十時からです
```

**5/5 identical, worst character error rate 0.000.** Synthetic speech is a
control, not a benchmark — it says the conversion transcribes what the fp32 model
transcribes, and nothing about word error rate on real speakers. (The fp32 model
itself mishears 明日 as 厚 on the last one; both builds mishear it the same way,
which is the point.)

```bash
python convert/check_wav2vec2.py fp32     # or int8
```

## The conversion note worth reading

The first build of this model was correct and **13x slower than it should have
been**: 3485 ms against eager's 215 ms, at correlation 1.000000. The cost was one
module.

wav2vec2 puts a positional convolution in front of its encoder,
`Conv1d(1024, 1024, kernel_size=128, groups=16)`, and it carries `weight_norm` —
a live `torch.nn.utils.parametrize` parametrization. Under `torch.export` that
makes the weight a *computed* tensor: the graph gets a `pow` and a `sum` where a
constant should be. XNNPACK's partitioner requires a weight that is a static
parameter, so it declines the convolution, and the convolution runs on the
portable kernels.

Measured on that module alone at sequence 499:

```
as loaded                          delegated  70.0%   3206.7 ms
after remove_parametrizations      delegated 100.0%      3.6 ms
```

with the output identical to the last bit. Materialising the weight before export
is what brought the whole model to 267.8 ms. It is reported upstream as
[pytorch/executorch#22078](https://github.com/pytorch/executorch/issues/22078),
because ExecuTorch already ships the pass that would fix it without touching the
model — it just does not run it during lowering.

- **Source**: [jonatasgrosman/wav2vec2-large-xlsr-53-japanese](https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-japanese)
- **License**: Apache-2.0

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))