Instructions to use espnet/frontia_vad_v1_online with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- ESPnet
How to use espnet/frontia_vad_v1_online with ESPnet:
unknown model type (must be text-to-speech or automatic-speech-recognition)
- Notebooks
- Google Colab
- Kaggle
espnet/frontia_vad_v1_online: voice activity detection, online
One of two ESPnet voice activity detectors trained together:
| model | reads ahead | for | |
|---|---|---|---|
| this one (online) | causal TCN | nothing: every frame is decided from the audio up to it | live streams and the microphone |
Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop)
gets the probability that someone is speaking. The model was trained as
one-speaker diarization (diar1, num_spk 1): every speaker of a recording
merged into one "speech" label.
Use
from espnet2.bin.diar_inference import VoiceActivityDetector
vad = VoiceActivityDetector.for_mode("online", "espnet/frontia_vad_v1_online")
for seg in vad.detect_file("recording.wav"):
print(f"{seg.start:.2f}\t{seg.end:.2f}")
From a terminal, espnet detect recording.wav (offline model) or
espnet detect --live (online model, from the microphone); from an agent, the
MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained
loads it too, as a num_spk=1 diarization model.
Decision rule
A frame is speech when its probability is above 0.7, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 0.
Latency: frame i covers samples [128 i, 128 i + 512), so its probability is ready 32 ms after the frame starts. The median filter then needs the frames after it: a median of 11 frames settles a decision 5 frames (40 ms) later. With median 1 the decision is final as soon as the probability is.
Results
Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.
| set | speech h | DetER (tuned) | FA | Miss | DetER (set's own best) | its threshold, median |
|---|---|---|---|---|---|---|
| aishell4_eval | 11.5 | 8.29 | 3.86 | 4.43 | 8.00 | 0.6, 11 |
| alimeeting_dev | 3.9 | 7.52 | 5.23 | 2.29 | 7.49 | 0.6, 11 |
| alimeeting_eval | 9.9 | 7.21 | 4.85 | 2.36 | 7.21 | 0.7, 11 |
| ami_dev | 75.1 | 10.17 | 4.78 | 5.39 | 9.79 | 0.6, 11 |
| ami_eval | 90.1 | 10.12 | 2.95 | 7.18 | 8.77 | 0.5, 11 |
| ami_mh_dev | 7.6 | 11.08 | 7.72 | 3.36 | 11.08 | 0.7, 11 |
| ami_mh_eval | 7.3 | 11.44 | 6.38 | 5.07 | 11.44 | 0.7, 11 |
| ava_speech_eval | 19.9 | 50.29 | 44.27 | 6.02 | 33.03 | 0.9, 11 |
| chime6_dev | 117.9 | 9.64 | 1.94 | 7.71 | 4.65 | 0.3, 11 |
| chime6_eval | 122.7 | 14.78 | 5.41 | 9.37 | 14.76 | 0.6, 11 |
| dev | 329.2 | 7.93 | 2.72 | 5.21 | 6.58 | 0.4, 11 |
| dipco_dev | 98.1 | 10.51 | 4.16 | 6.35 | 8.52 | 0.4, 11 |
| dipco_eval | 92.0 | 10.06 | 5.38 | 4.69 | 9.41 | 0.6, 11 |
| earnings21_eval | 32.8 | 14.87 | 10.51 | 4.36 | 14.87 | 0.7, 11 |
| icsi_dev | 12.5 | 11.96 | 5.42 | 6.55 | 11.96 | 0.7, 11 |
| icsi_eval | 11.9 | 14.71 | 3.92 | 10.79 | 13.60 | 0.5, 11 |
| notsofar_dev | 123.7 | 4.52 | 1.94 | 2.59 | 4.37 | 0.6, 11 |
| notsofar_eval | 398.5 | 4.48 | 2.87 | 1.62 | 4.48 | 0.7, 11 |
| ten_vad_eval | 0.1 | 22.97 | 21.46 | 1.51 | 16.92 | 0.9, 11 |
| voxconverse_dev | 18.9 | 4.94 | 3.89 | 1.05 | 4.94 | 0.7, 11 |
| voxconverse_eval | 39.0 | 8.26 | 7.32 | 0.94 | 8.08 | 0.8, 11 |
Hour-weighted DetER: 8.71. Macro average over the 21 sets: 12.18.
Training
Recipe: egs2/frontia_vad_v1/diar1 in ESPnet.
The config is in this repository. Weights: the checkpoint of epoch 11.
| training data | licence |
|---|---|
| AMI (every microphone) | CC BY 4.0 |
| CHiME-6, OpenSLR SLR150 (every array and binaural channel) | CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0) |
| NOTSOFAR-1 240825.1 | CC BY 4.0 |
| ICSI | CC BY 4.0 |
noise: MUSAN noise/ and music/; CHiME-3 background |
CC licences (some music/ tracks CC BY-SA) / public domain; CC BY 4.0 |
The other sets in the table above are used for evaluation only.
Training noise
The training audio was augmented by adding recorded noise to it. That noise
audio is not distributed with this model; it was used only as additive
background noise during training. All of MUSAN music/ was used, and some of
those tracks are licensed CC BY-SA. The next release restricts the music to
MUSAN tracks without vocals that are CC0, CC BY or public domain.
Licence
CC BY-SA 4.0. The training data include material under CC BY-SA: CHiME-6 and
some MUSAN music/ tracks.
- Downloads last month
- 64