dys-asr/parakeet-rnnt-0.6b-v2split-sampler

Parakeet RNN-T 0.6B — etiology-weighted sampling

Identical to dys-asr/parakeet-rnnt-0.6b-v2split-uniform in every respect but one: each training utterance is drawn with probability proportional to its etiology's share of a target distribution, rather than uniformly. The two runs exist to measure that one change.

It did not work. On the held-out split the weighted sampler is 0.13 CER points worse than uniform. This model is published because a negative result that is not published gets repeated.

The sampler

$$p_d\propto q_d{\left(\frac{E_d}{\bar{E}}\right)}^{0.5},w_i=\frac{p_{d(i)}}{N_{d(i)}}p\_d\; \backslash propto\; q\_d\; \backslash left(\backslash frac\{E\_d\}\{\backslash bar\; E\}\backslash right)^\{0.5\},\; \backslash qquad\; w\_i\; =\; \backslash frac\{p\_\{d(i)\}\}\{N\_\{d(i)\}\}$$

• $q_d$ — the etiology's share of the private test set by character mass (its speaker count times its characters per speaker). The metric pools characters, so a speaker prior would not align with its denominator; measured on this corpus, characters per speaker range from 11,222 for Stroke to 32,338 for Parkinson's, a factor of 2.9.
• $E_d$ — the median per-speaker CER the uniform run reaches on the same held-out split. Median rather than mean because every etiology's speaker CERs are strongly right-skewed: Down Syndrome averages 12.92% against a median of 8.45% because one of its seventeen speakers sits at 57.71%, and a mean would aim the sampler at a handful of outliers instead of at the group.
• $\beta = 0.5$ damps the difficulty term, so an etiology twice as hard is sampled $\sqrt2$ times more rather than twice as much.
• Dividing by $N_d$ turns a group share into a per-utterance weight. Without it a group would be up-weighted twice, once by its share and again by however many utterances it happens to hold.

Drawn with replacement, one draw per epoch, the same draw on every rank so that accelerate's sharding stays consistent. The epoch length is unchanged, so the learning-rate schedule is identical to the uniform run's.

etiology	median CER (uniform run)	share of a sampled epoch	per-utterance, vs uniform
Cerebral Palsy	10.36%	41.9%	1.74x
Down Syndrome	8.45%	20.7%	1.30x
ALS	1.37%	19.5%	1.09x
Stroke	7.22%	9.2%	1.04x
Parkinson's Disease	2.63%	8.7%	0.26x

No floor was applied, deliberately.

Results

Held-out split, 38,141 utterances after filtering:

	best epoch	4,000-utterance subset	full split
uniform control	9	4.84%	4.94%
this model	10	4.92%	5.07%

These are dev numbers on a locally held-out split and are not comparable to the CERs this family reports from the challenge's sequestered test set.

Why it probably lost

The competition scores pooled CER — total character errors over total reference characters — while the difficulty term pulls the sampler towards equalising error across etiologies. Moving budget away from ALS, where the model is already at 1.37% median CER, costs more than it buys: ALS is 36.6% of the test set by character mass, so a small regression there outweighs a larger proportional gain on Cerebral Palsy.

Parkinson's took the sharpest cut, to 0.26x, and it is the largest group in the corpus at 146,653 utterances. This run saw roughly a quarter of the Parkinson's exposure the control did, and its dev CER was still falling at epoch 10 (5.13% -> 4.92%) where the control had flattened (4.84% -> 4.84%). Longer training might close the gap, but that would no longer be the same comparison.

One seed. The gap is about seven times the sampling standard error on the full split, so it is unlikely to be noise, but a single run cannot separate the sampler from the seed.

Training data

source	records
SAPC-1 train and dev	250,014
SAPC-2 train, less SAPC1 test1	132,494
synthetic CosyVoice3 speech	55,988
chunks force-aligned out of over-length recordings	8,278
total before filtering	446,774

441,928 survive the 0.5-45 s and 200-label-token filters. Nothing comes from outside the challenge corpora, so this is a constrained-track model.

SAPC1's test1 — a 50-speaker official split that SAPC2 re-released inside its own training data — is carved back out and added to SAPC2 dev, because SAPC2's own dev split contains no Parkinson's speakers at all and a per-etiology measurement on it would be blind to the largest group in the corpus. Speaker, file-path and basename overlap with the training manifests are all zero.

Training procedure

Ten epochs on sixteen GH200s, effective batch 32 at per-device 2, AdamW at 1e-4 with a tri-stage schedule (10% warmup, 40% hold), weight decay 0.01, layerdrop 0.05, gradient clip 1.0, bf16, seed 42, 138,110 optimizer steps. Augmentation is speed perturbation over 0.8-1.2, SpecAugment (5% of the time axis in spans of 10 frames, 40% of the mel axis in spans of 27 bins) and SpecCutout (two 20x20 rectangles). The published weights are epoch 10, the best of ten by dev CER.

Usage

import soundfile as sf
import torch
from transformers import AutoProcessor, ParakeetForRNNT

model_id = "dys-asr/parakeet-rnnt-0.6b-v2split-sampler"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(model_id)
model = ParakeetForRNNT.from_pretrained(model_id).to(device).eval()

audio, rate = sf.read("utterance.wav", dtype="float32")
inputs = processor(audio, sampling_rate=rate, return_tensors="pt").to(device)
with torch.inference_mode():
    generated = model.generate(**inputs)
print(processor.batch_decode(generated, skip_special_tokens=True)[0])

Requires transformers>=5.9. Audio must be 16 kHz mono.

Intended use and limitations

A competition entry for the Speech Accessibility Project Challenge, and research on atypical-speech recognition.

• Worse than its uniform control on held-out data. Use the control unless you are studying the sampler itself.
• Single run, single seed. No variance estimate.
• Requires transformers>=5.9. Not loadable by the 4.x line.
• Slow relative to CTC. Decoding is autoregressive.
• Lower-case, unpunctuated, numerals written as words.
• English only, 16 kHz mono.
• Not a clinical tool. Nothing here supports inference about any diagnosis.

License and attribution

The Speech Accessibility Project corpora are governed by their own data use agreement and are not redistributed here; reproducing this training set requires authorized access. Fun-CosyVoice3's terms apply to the synthesis component.