JacobLinCool/accent-detective

Accent Detective

Accent Detective scores how much a Mandarin recording sounds like Taiwan Mandarin. It is one network that can be cut at four depths, so the same weights serve anything from a small CPU deployment to the most accurate setting. The text the recording reads is optional: without it the model judges the audio alone, and with it the model also looks at the syllables the text implies. Neither mode needs a forced alignment.

中文說明。 評估一段華語錄音聽起來有多像臺灣華語。這是一個網路，可以在四個深度截斷使用：從資源有限的 CPU 環境，到最準的設定，都用同一組權重。錄音念的文字可以不提供。沒有文字時，只看語音；有文字時，也會參考文字對應的音節。兩種方式都不需要強制對齊。

What it measures

The model listens to how Mandarin is pronounced: tone contours, neutral tones, initials and finals, and rhythm. It does not look at word choice, and it does not check whether a reading is "correct". During training, a text-only model absorbs whatever the sentence content alone predicts, so the score reflects how the sentence is spoken, not which words it contains.

It was built to evaluate text-to-speech systems: comparing different renditions of the same sentence. It is not meant to classify people.

• Score: calibrated log-odds that the recording is Taiwan Mandarin, probability the same as a probability, and index the log-odds on a 0–1 scale (0.5 is undecided, each 0.1 is about 7× the odds).
• Calibration: one Platt transform per size and mode, fitted on development data, ships with the weights.

Usage

# pip install -r requirements.txt   (torch, torchaudio, librosa, soundfile, numpy, pypinyin, jieba, opencc, safetensors)
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("JacobLinCool/accent-detective")
sys.path.insert(0, path)
from detective import load

det = load()                                          # CPU or GPU
det.score("clip.wav")                                 # no text, the largest size: {"logit", "probability", "index", "size", "mode", ...}
det.score("clip.wav", "這麼長的時間", size="tiny")      # with the text it reads, size tiny
det.score_sizes("clip.wav")                           # all four sizes from one pass
det.score_recording("talk.wav")                       # any length: long audio is cut at pauses into 4–10 s pieces

• Sizes: nano, tiny, mini and small, smallest first. A smaller size is the first part of a larger one, so score_sizes costs one pass of the largest size.
• Text: optional. If it has no Han syllables, the audio mode is used. Recordings longer than 15 s are scored in pieces without the text, because the text cannot be split without an alignment.

Model

Everything is one checkpoint, model.safetensors, of 53.9 M parameters.

• Trunk: a Conformer over log-mel frames (width 384, 18 layers, 20 ms frames), with exits after layers 3, 6, 9 and 18. Each exit predicts hidden layers of two self-supervised teachers: layers 4, 8 and 12 of TencentGameMate/chinese-wav2vec2-base, and layers 8 and 12 of TencentGameMate/chinese-hubert-base.
• Audio readout (one per exit): the predicted teacher layers, with their utterance mean removed, pass through attentive statistics pooling into an utterance log-odds.
• Text readout (one per exit): each syllable of the text becomes a query. The query encodes the syllable, its Mainland and Taiwan tones, tone sandhi, neutral tone and position. Two attention layers read the audio readout's frames under a soft monotonic position prior, and the mean of the syllable outputs is added to the audio log-odds as a correction.

Size	Trunk layers	Parameters, audio only	Parameters, with text
Nano	3	9.5 M	10.5 M
Tiny	6	16.2 M	17.2 M
Mini	9	22.9 M	23.9 M
Small	18	43.0 M	44.0 M

Training. All four exits are trained at once:

• Distillation: feature distillation from both teachers.
• Accent task: three signals, each exit's coefficient set once from the ratio of gradient norms at the start of training.
  • soft targets from an ensemble of HuBERT-based accent judges, on speakers outside every evaluation set
  • ranking of same-sentence Taiwan and Mainland recordings
  • consistency of the score under simulated channel changes: noise, band-limiting, µ-law

The readouts are then trained on the frozen trunk. Each uses importance weights for sentence length and text context, plus a product-of-experts bias model that sees only the text, so that the readouts do not learn from content.

Evaluation

All numbers come from scoring the audio end to end with the released checkpoint. Real speech is a sealed set of Common Voice speakers whose labels were never used in training or development, and who were used only for reporting, never to choose a design or a setting. Their recordings were not entirely unheard: the Taiwan clips, and 69 of the Mainland clips, were among the unlabelled audio on which an earlier, smaller student was distilled, and the released trunk starts from that student. No other system in the comparisons heard them.

	Nano	Tiny	Mini	Small
Length-matched AUC, sealed speakers, listener-checked labels (583 Taiwan, 1,451 Mainland clips, 448 speakers)	0.956	0.961	0.960	0.962
The same, with the text	0.956	0.960	0.960	0.962
The same, Common Voice labels as published (2,060 clips)	0.888	0.903	0.905	0.913
Expected calibration error, sealed speakers, listener-checked labels	0.031	0.028	0.028	0.034
Same text and voice prompt, BreezyVoice ranked above CosyVoice-300M (1,000 pairs)	79%	79%	80%	80%
The same contrast in standard deviations of the score on real speech	0.38	0.36	0.37	0.37
A comparison of the two synthesisers on 10 random pairs that ranks BreezyVoice first	99%	99%	99%	99%
Score variance explained by the TTS voice (119 texts × 4 voices of one TTS)	6.8%	4.8%	5.5%	11.9%
CPU latency for a 5 s clip, audio only, 4 threads / 1 thread	29 / 74 ms	51 / 132 ms	73 / 192 ms	133 / 355 ms
CPU latency with the text, 4 threads / 1 thread	35 / 81 ms	57 / 140 ms	78 / 200 ms	139 / 362 ms

• Listener-checked labels: a Common Voice label records the language version a volunteer chose, not their accent, and some clips hold no speech or no Mandarin.
  • One listener heard, blind, the 248 sealed clips that at least half of twelve strong systems got wrong, plus 40 clips that all of them got right.
  • Clips without speech or not in Mandarin were removed. Clips that leaned the other way were relabelled (100). Clips the listener could not tell keep their label.
  • On the disputed clips the listener confirmed only one label in five. On the clips all systems got right, they disagreed with a few labels too, so the checked-label AUC is an upper estimate. The comparison between systems is the reliable part.
• Reference: on the same clips, an ensemble of three HuBERT-base accent judges (about 400 M parameters in use, with a forced aligner) reaches 0.964 AUC with listener-checked labels and 0.918 with the published labels.
• Synthetic contrast: the 1,000 pairs are 250 Common Voice zh-TW sentences, each synthesised with four voice prompts.
• Voice set: the four voices come from a single TTS model, so the voice share has a wide interval (8–17% for Small) and may include real differences between the voices.
• Latency: measured on a cloud x86 CPU (Google Colab), batch 1, fp32, median of 20 runs. It includes turning the text into syllables.

Recording conditions. Change in length-matched AUC on the sealed speakers, audio only, listener-checked labels. MP3, real noise and real rooms were never used in training. Training did use simulated band-limiting and µ-law, which are close to the telephone condition.

Condition	Nano	Tiny	Mini	Small
MP3 at 32 kbit/s	−0.003	−0.003	−0.003	−0.001
Telephone (8 kHz, G.711 µ-law)	−0.029	−0.030	−0.031	−0.019
Real point-source noise at 10 dB SNR (OpenSLR 28)	−0.055	−0.045	−0.040	−0.031
Real room reverberation (OpenSLR 28)	−0.040	−0.030	−0.028	−0.029

Limitations

• The text adds little: once content is controlled, the text mode is as accurate as the audio mode, not more, at every size.
• Southwestern Mandarin: standard Mandarin from speakers of Sichuan, Chongqing, Guizhou and Yunnan (KeSpeech) is judged Taiwan-accented 20–32% of the time, depending on the size. Standard Mandarin from the other Mainland regions is judged Taiwan-accented 2–8% of the time.
• Noise and reverberation: they lower accuracy, most for the smallest size (see the table above).
• Not a language detector: a clip without speech, or in another language, still gets a score. Check that the audio is Mandarin speech first.
• Training domain: volunteer read speech from Common Voice.
• Not evaluated: Hong Kong, Singapore and Malaysian Mandarin.

Intended use

Research, and evaluating or steering speech synthesis towards Taiwan Mandarin. It scores speech, not people: do not use it to infer anyone's nationality or origin.

Training data and licences

• Labelled data: Common Voice 25 zh-TW and Common Voice 21 zh-CN, both CC0, with speakers disjoint between training, development and the sealed evaluation set.
• Distillation audio: about 106 h, used without labels:
  • 21 h of the Common Voice training audio
  • 60 h of AISHELL-1 and 25 h of THCHS-30, both Apache 2.0
• Evaluation only: KeSpeech, and the OpenSLR 28 noises and room impulse responses (Apache 2.0).
• Code and weights: MIT (see LICENSE).
• Distillation teachers: TencentGameMate/chinese-wav2vec2-base and TencentGameMate/chinese-hubert-base, both MIT.
• Bundled lexicons:
  • lexicon/concised.json.xz: headwords and their pinyin extracted from 教育部《國語辭典簡編本》 (Ministry of Education, Taiwan), CC BY-ND 3.0 TW, converted to JSON.
  • lexicon/cedict.txt.gz: CC-CEDICT (MDBG), CC BY-SA 4.0.

Citation

@misc{lin2026accentdetective,
  title        = {Accent Detective: A Compact Judge of Taiwan Mandarin Accent},
  author       = {Lin, Jhen-Ke},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/JacobLinCool/accent-detective}},
  note         = {Hugging Face model}
}