espnet/frontia_vad_v1_offline: voice activity detection, offline

One of two ESPnet voice activity detectors trained together:

model reads ahead for
this one (offline) dual-mode TCN, run at its largest look-ahead up to 124 frames (0.99 s) of future audio whole recordings, cleaning a corpus

Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop) gets the probability that someone is speaking. The model was trained as one-speaker diarization (diar1, num_spk 1): every speaker of a recording merged into one "speech" label.

Use

from espnet2.bin.diar_inference import VoiceActivityDetector

vad = VoiceActivityDetector.for_mode("offline", "espnet/frontia_vad_v1_offline")
for seg in vad.detect_file("recording.wav"):
    print(f"{seg.start:.2f}\t{seg.end:.2f}")

From a terminal, espnet detect recording.wav (offline model) or espnet detect --live (online model, from the microphone); from an agent, the MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained loads it too, as a num_spk=1 diarization model.

Decision rule

A frame is speech when its probability is above 0.8, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 124 (the model's max_look_ahead).

Latency: a frame's probability reads 124 frames (0.99 s) of future audio, and the median filter 5 frames (40 ms) more, so this model is for whole recordings rather than live streams.

Results

Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.

set speech h DetER (tuned) FA Miss DetER (set's own best) its threshold, median
aishell4_eval 11.5 7.51 2.11 5.40 5.70 0.5, 11
alimeeting_dev 3.9 6.26 4.00 2.26 6.21 0.7, 11
alimeeting_eval 9.9 5.77 3.37 2.40 5.77 0.8, 11
ami_dev 75.1 8.48 3.39 5.09 7.75 0.6, 11
ami_eval 90.1 8.63 1.70 6.93 6.73 0.6, 11
ami_mh_dev 7.6 8.89 5.38 3.51 8.89 0.8, 11
ami_mh_eval 7.3 9.75 4.10 5.65 9.33 0.7, 11
ava_speech_eval 19.9 32.39 22.59 9.80 28.47 0.9, 11
chime6_dev 117.9 9.75 1.58 8.18 4.25 0.3, 11
chime6_eval 122.7 13.92 3.57 10.35 12.54 0.6, 11
dev 329.2 7.12 1.77 5.35 5.27 0.5, 11
dipco_dev 98.1 11.67 2.64 9.04 7.67 0.5, 11
dipco_eval 92.0 9.96 3.56 6.41 8.53 0.6, 11
earnings21_eval 32.8 9.50 6.94 2.56 9.50 0.8, 11
icsi_dev 12.5 11.03 2.83 8.20 10.35 0.7, 11
icsi_eval 11.9 15.16 2.25 12.91 12.67 0.5, 11
notsofar_dev 123.7 3.39 0.87 2.52 2.89 0.7, 11
notsofar_eval 398.5 3.24 1.58 1.65 3.15 0.7, 11
ten_vad_eval 0.1 19.52 17.53 1.98 16.97 0.9, 11
voxconverse_dev 18.9 4.54 2.58 1.96 4.48 0.7, 11
voxconverse_eval 39.0 7.42 5.72 1.70 7.42 0.8, 11

Hour-weighted DetER: 7.61. Macro average over the 21 sets: 10.19.

Training

Recipe: egs2/frontia_vad_v1/diar1 in ESPnet. The config is in this repository. Weights: the average of the two checkpoints with the best validation accuracy, epochs 11 and 17.

training data licence
AMI (every microphone) CC BY 4.0
CHiME-6, OpenSLR SLR150 (every array and binaural channel) CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0)
NOTSOFAR-1 240825.1 CC BY 4.0
ICSI CC BY 4.0
noise: MUSAN noise/ and music/; CHiME-3 background CC licences (some music/ tracks CC BY-SA) / public domain; CC BY 4.0

The other sets in the table above are used for evaluation only.

Training noise

The training audio was augmented by adding recorded noise to it. That noise audio is not distributed with this model; it was used only as additive background noise during training. All of MUSAN music/ was used, and some of those tracks are licensed CC BY-SA. The next release restricts the music to MUSAN tracks without vocals that are CC0, CC BY or public domain.

Licence

CC BY-SA 4.0. The training data include material under CC BY-SA: CHiME-6 and some MUSAN music/ tracks.

Downloads last month
67
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support