mlboydaisuke/wav2vec2-XLSR53-Japanese-ExecuTorch

wav2vec2 XLSR-53 Japanese — ExecuTorch (CTC speech recognition)

Ten seconds of Japanese audio in, a transcript out, in one forward pass. No decoder loop, no beam, no KV cache, nothing to carry between calls — the encoder is the model and the transcript falls out of an argmax per 20 ms frame.

That is the difference from the Whisper conversions on this shelf, which are encoder-decoder: one encoder pass per window plus one decoder pass per generated token. A CTC model trades that machinery — and the ability to punctuate — for a much simpler thing to run on a device.

(waveform (1,160000) fp32 16 kHz mono, attention_mask (1,160000) int64)
    -> logits (1, 499, 2341)

build	file	MB	corr vs fp32 eager	transcripts identical	Mac ms*
fp32	wav2vec2_xlsr53_ja_xnnpack_fp32.pte	1271.7	1.000000	5/5	267.8
int8 (dynamic)	wav2vec2_xlsr53_ja_xnnpack_int8.pte	357.9	0.997838	5/5	228.9

*Mac arm64, single process, median of 5, for a 10-second window — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine: 215.1 ms. XNNPACK delegate coverage 64.5% (869/1347 ops) for fp32, 75.2% for int8.

Take the int8 build. It is 28% of the file, it is faster, and it returns the same five transcripts character for character, at worst character error rate 0.000. Its correlation is 0.9978 rather than 1.000000, and on this task that difference did not reach the output — but five sentences of synthetic speech is what stands behind that, not a benchmark.

There is no Core ML build. coremltools rejects the graph on a stack whose operands are a mix of int and float. This one is XNNPACK only, and the same file runs on Android and on the Mac.

Running it

1. The audio. 16 kHz mono in [-1, 1], exactly 160 000 samples. Zero-pad a shorter clip; cut a longer one. Do not normalise it — the per-utterance normalisation wav2vec2 expects is inside the graph, and it needs the mask to be correct (below).

2. The mask. attention_mask is int64 (1, 160000), 1 on the real samples and 0 on the padding. It is not optional and it is not cosmetic: padding a 3.7-second clip to 10 seconds without it changed the transcript on 4 of 5 test sentences.

with the mask:     今日はよい天気ですね
without:           今わよい電気で住

The zeros drag the mean and shrink the variance the normalisation divides by, so the speech comes out at the wrong scale, and the encoder attends to the silence. With the mask — masked mean and variance, and the same mask handed to the encoder — all five come back identical to the transcript of the unpadded clip.

3. The decode. CTC greedy, over logits[0]:

ids = argmax per frame
collapse runs of the same id
drop the blank (id 0)
map through the repo's vocab.json, turn the word delimiter | into a space, strip

That last clause matters. Without the strip this recipe returns '今日はよい天気ですね ' where the model's own tokenizer returns '今日はよい天気ですね' — a | lands at the end of the utterance. The checker in the conversion repo runs the hand-written recipe and asserts it against processor.batch_decode on the same ids, which is how that was caught.

There is no punctuation and no casing in the output: the vocabulary has 2341 entries and none of them are 。 or 、. The model also writes numbers as digits (コーヒーを1杯).

Verification

Five sentences spoken by macOS say -v Kyoko, transcribed through the .pte and through the eager model on the identical padded window and mask:

今日はよい天気ですね
駅までの道を教えてください
この電車は東京駅に泊まりますか
コーヒーを1杯お願いします
厚の会議は十時からです

5/5 identical, worst character error rate 0.000. Synthetic speech is a control, not a benchmark — it says the conversion transcribes what the fp32 model transcribes, and nothing about word error rate on real speakers. (The fp32 model itself mishears 明日 as 厚 on the last one; both builds mishear it the same way, which is the point.)

python convert/check_wav2vec2.py fp32     # or int8

The conversion note worth reading

The first build of this model was correct and 13x slower than it should have been: 3485 ms against eager's 215 ms, at correlation 1.000000. The cost was one module.

wav2vec2 puts a positional convolution in front of its encoder, Conv1d(1024, 1024, kernel_size=128, groups=16), and it carries weight_norm — a live torch.nn.utils.parametrize parametrization. Under torch.export that makes the weight a computed tensor: the graph gets a pow and a sum where a constant should be. XNNPACK's partitioner requires a weight that is a static parameter, so it declines the convolution, and the convolution runs on the portable kernels.

Measured on that module alone at sequence 499:

as loaded                          delegated  70.0%   3206.7 ms
after remove_parametrizations      delegated 100.0%      3.6 ms

with the output identical to the last bit. Materialising the weight before export is what brought the whole model to 267.8 ms. It is reported upstream as pytorch/executorch#22078, because ExecuTorch already ships the pass that would fix it without touching the model — it just does not run it during lowering.

• Source: jonatasgrosman/wav2vec2-large-xlsr-53-japanese
• License: Apache-2.0

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)