Accent Detective v2: student distilled on more varied audio (AISHELL-1, THCHS-30); evaluated on held-out speakers

README.md

@@ -73,22 +73,25 @@ These keep the judges from learning from content.
 
 ## Evaluation
 
-Compared with the 411 M-parameter pipeline it replaces (MMS alignment + 12-layer wav2vec 2.0 + judge):
+Compared with the 411 M-parameter pipeline it replaces (MMS alignment + 12-layer wav2vec 2.0 + one judge, mean of three seeds). Every system scores the audio end to end.
 
 | | Original | Accent Detective |
 |---|---:|---:|
 | Parameters | 411 M | 18.6 M |
-| Length-matched AUC, unseen speakers (Common Voice) | 0.969 | 0.966 |
-| Same text and voice prompt, BreezyVoice ranked above CosyVoice-300M | 84% | 84% |
-| Score variance explained by the TTS voice (120 texts × 4 voices) | 6.3% | 5.0% |
-| Expected calibration error, no recalibration | 0.073 | 0.062 |
-| CPU latency per utterance (Apple M-series, 6 threads, median) | 291 ms | 17 ms |
+| Length-matched AUC, held-out Common Voice speakers (526 Taiwan, 704 Mainland) | 0.949 | 0.940 [0.909, 0.959] |
+| Same text and voice prompt, BreezyVoice ranked above CosyVoice-300M (1,000 pairs) | 72% | 75% |
+| The same contrast in standard deviations of the score on real speech | 0.31 | 0.40 |
+| Score variance explained by the TTS voice (119 texts × 4 voices) | 6.3% | 6.9% |
+| Expected calibration error, no recalibration | 0.051 | 0.034 |
+| CPU latency per utterance (Apple M5 Pro, 6 threads, median) | 184 ms | 16 ms |
 
-The configuration was selected on the test set and is also the best on the development set. On an NVIDIA L4 with CUDA graphs, a lighter variant (transformer student, one judge) runs in 4.0 ms, against 46 ms for the original pipeline.
+- **Held-out set:** the held-out speakers were never used to train or choose any model. Their sentences are harder than the original test set's: every system scores lower on them.
+- **Synthetic contrast:** the 1,000 pairs are 250 held-out Taiwan sentences read with four voice prompts.
 
 ## Limitations
 
-- **Southwestern Mandarin:** speakers from Sichuan, Chongqing, Guizhou and Yunnan are judged Taiwanese about a third of the time when reading standard Mandarin. Other Mainland regions rarely are. The overlap is largest on retroflex initials and velar nasal finals, which both varieties merge.
+- **Southwestern Mandarin:** standard Mandarin from speakers of Sichuan, Chongqing, Guizhou and Yunnan (KeSpeech) is judged Taiwan-accented 37% of the time. The overlap is largest on retroflex initials and velar nasal finals, which both varieties merge.
+- **Other recording conditions:** on KeSpeech, standard Mandarin from the other Mainland regions is judged Taiwan-accented 3–16% of the time (11% on average), against about 5% for the original pipeline. The compact model is less robust than its teacher to recording conditions unlike Common Voice.
 - **Training domain:** volunteer read speech from Common Voice.
 - **Not evaluated:** Hong Kong, Singapore and Malaysian Mandarin.
 
@@ -98,7 +101,8 @@ Research, and evaluating or steering speech synthesis towards Taiwan Mandarin. I
 
 ## Training data and licences
 
-- **Training data:** Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint across splits. The student was distilled on about 90 h of the same public Common Voice audio, excluding every development and test speaker.
+- **Training data:** Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint across splits.
+- **Distillation audio:** about 140 h, used without labels. It holds 84 h of the same public Common Voice audio, excluding every development, test and held-out speaker, and 55 h of AISHELL-1 and THCHS-30 (Apache 2.0) for more varied recording conditions.
 - **Code and weights:** MIT (see `LICENSE`).
 - **Distillation teacher:** TencentGameMate/chinese-wav2vec2-base (MIT).
 - **Bundled lexicons:**

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ab6d6189ac1053154f05fd9ea373f323a5f5da1f0e7ed06d90e98efaa9018408
-size 74303292
+oid sha256:b1fca94d503c2b1c77a687fecb0684e8af982d6388925fe80da97bdb83b99178
+size 74303340

twjudge/accent/augment.py

@@ -0,0 +1,82 @@
+"""Recording-condition augmentation: additive noise, room reverberation, telephone band, 8-bit mu-law, gain.
+
+Used to make the student encoder (distillation) and the judges (training) less sensitive to the
+recording channel, the weakness found on KeSpeech. Every class gets the same augmentation, so the
+channel never becomes a label cue. Assets are packed once by `judge/scripts/augment_assets.py`:
+
+  noise.npy / noise_off.npy   MUSAN noise and music (CC BY 4.0), int16 16 kHz, concatenated
+  rir.npy                     OpenSLR 28 room impulse responses (Apache 2.0), float32 (n, L), peak-aligned
+
+The whole batch is processed at once on the device; band limits are applied as smooth masks on the
+spectrum (no recursive filters, which are slow on a GPU).
+
+    aug = Augmenter("assets/aug", device="cuda")
+    x = aug(x)          # (B, T) float waveforms on the device, each item augmented independently
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+
+SR = 16000
+
+
+class Augmenter:
+    def __init__(self, assets: str | Path, device: str = "cpu", p: float = 0.8, seed: int = 0):
+        import torch
+
+        d = Path(assets)
+        self.noise = torch.from_numpy(np.load(d / "noise.npy").astype(np.float32) / 32768).to(device)
+        self.noise_off = np.load(d / "noise_off.npy")
+        self.rir = torch.from_numpy(np.load(d / "rir.npy")).to(device)
+        self.dev, self.p = device, p
+        self.rng = np.random.default_rng(seed)
+
+    def _noise(self, n: int):
+        k = self.rng.integers(len(self.noise_off) - 1)
+        a, b = int(self.noise_off[k]), int(self.noise_off[k + 1])
+        if b - a <= n:
+            return self.noise[a:b].repeat(n // (b - a) + 1)[:n]
+        o = self.rng.integers(a, b - n)
+        return self.noise[o:o + n]
+
+    def __call__(self, x):
+        """x: (B, T) or (T,) float tensor on the device; each item is augmented with probability p."""
+        import torch
+        import torchaudio.functional as AF
+
+        one = x.dim() == 1
+        x = x[None] if one else x.clone()
+        Bn, T = x.shape
+        u = lambda: self.rng.random(Bn)
+        on = u() < self.p
+        rev, noi, band, mu = on & (u() < 0.35), on & (u() < 0.6), np.where(on, u(), 1.0), on & (u() < 0.15)
+        if rev.any():                                                 # reverberation
+            i = torch.from_numpy(np.where(rev)[0]).to(x.device)
+            h = self.rir[torch.from_numpy(self.rng.integers(len(self.rir), size=len(i))).to(x.device)]
+            y = AF.fftconvolve(x[i], h)[:, :T]
+            x[i] = y * (x[i].abs().amax(1, keepdim=True) / (y.abs().amax(1, keepdim=True) + 1e-6))
+        if noi.any():                                                 # noise at 5–25 dB SNR
+            i = torch.from_numpy(np.where(noi)[0]).to(x.device)
+            nz = torch.stack([self._noise(T) for _ in range(len(i))])
+            snr = torch.from_numpy(self.rng.uniform(5, 25, len(i))).float().to(x.device)
+            px, pn = x[i].pow(2).mean(1) + 1e-8, nz.pow(2).mean(1) + 1e-8
+            x[i] = x[i] + nz * torch.sqrt(px / (pn * 10 ** (snr / 10)))[:, None]
+        tel, low = band < 0.25, (band >= 0.25) & (band < 0.4)
+        if tel.any() or low.any():                                    # telephone band / cheap microphone
+            f = torch.fft.rfftfreq(T, 1 / SR, device=x.device)[None]
+            lo = np.where(tel, 300.0, 0.0)
+            hi = np.where(tel, 3400.0, np.where(low, self.rng.uniform(3000, 6000, Bn), SR))
+            lo_t, hi_t = (torch.from_numpy(v).float().to(x.device)[:, None] for v in (lo, hi))
+            g = torch.sigmoid((f - lo_t) / 40) * torch.sigmoid((hi_t - f) / 150)
+            i = torch.from_numpy(np.where(tel | low)[0]).to(x.device)
+            x[i] = torch.fft.irfft(torch.fft.rfft(x[i]) * g[i], n=T)
+        if mu.any():                                                  # 8-bit mu-law
+            i = torch.from_numpy(np.where(mu)[0]).to(x.device)
+            peak = x[i].abs().amax(1, keepdim=True) + 1e-6
+            x[i] = AF.mu_law_decoding(AF.mu_law_encoding(x[i] / peak, 256), 256) * peak
+        gain = torch.from_numpy(np.where(on, 10 ** (self.rng.uniform(-10, 6, Bn) / 20), 1.0)).float().to(x.device)
+        x = torch.nan_to_num(x * gain[:, None]).clamp(-1, 1)
+        return x[0] if one else x