Instructions to use espnet/frontia_vad_v1_offline with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- ESPnet
How to use espnet/frontia_vad_v1_offline with ESPnet:
unknown model type (must be text-to-speech or automatic-speech-recognition)
- Notebooks
- Google Colab
- Kaggle
espnet/frontia_vad_v1_offline: voice activity detection, offline
One of two ESPnet voice activity detectors trained together:
| model | reads ahead | for | |
|---|---|---|---|
| this one (offline) | dual-mode TCN, run at its largest look-ahead | up to 124 frames (0.99 s) of future audio | whole recordings, cleaning a corpus |
Each 8 ms frame (16 kHz, 512-point STFT without centring, 128-sample hop)
gets the probability that someone is speaking. The model was trained as
one-speaker diarization (diar1, num_spk 1): every speaker of a recording
merged into one "speech" label.
Use
from espnet2.bin.diar_inference import VoiceActivityDetector
vad = VoiceActivityDetector.for_mode("offline", "espnet/frontia_vad_v1_offline")
for seg in vad.detect_file("recording.wav"):
print(f"{seg.start:.2f}\t{seg.end:.2f}")
From a terminal, espnet detect recording.wav (offline model) or
espnet detect --live (online model, from the microphone); from an agent, the
MCP tool detect of espnet-mcp. DiarizeSpeech.from_pretrained
loads it too, as a num_spk=1 diarization model.
Decision rule
A frame is speech when its probability is above 0.8, and the 0/1 sequence is then median-filtered over 11 frames: the setting with the lowest detection error on AMI MH dev. Look-ahead used: 124 (the model's max_look_ahead).
Latency: a frame's probability reads 124 frames (0.99 s) of future audio, and the median filter 5 frames (40 ms) more, so this model is for whole recordings rather than live streams.
Results
Detection error (%) = (missed + false-alarm speech) / reference speech, md-eval with a UEM over each whole recording, collar 0, at the rule above; the last two columns are each set's own best setting, for reference.
| set | speech h | DetER (tuned) | FA | Miss | DetER (set's own best) | its threshold, median |
|---|---|---|---|---|---|---|
| aishell4_eval | 11.5 | 7.51 | 2.11 | 5.40 | 5.70 | 0.5, 11 |
| alimeeting_dev | 3.9 | 6.26 | 4.00 | 2.26 | 6.21 | 0.7, 11 |
| alimeeting_eval | 9.9 | 5.77 | 3.37 | 2.40 | 5.77 | 0.8, 11 |
| ami_dev | 75.1 | 8.48 | 3.39 | 5.09 | 7.75 | 0.6, 11 |
| ami_eval | 90.1 | 8.63 | 1.70 | 6.93 | 6.73 | 0.6, 11 |
| ami_mh_dev | 7.6 | 8.89 | 5.38 | 3.51 | 8.89 | 0.8, 11 |
| ami_mh_eval | 7.3 | 9.75 | 4.10 | 5.65 | 9.33 | 0.7, 11 |
| ava_speech_eval | 19.9 | 32.39 | 22.59 | 9.80 | 28.47 | 0.9, 11 |
| chime6_dev | 117.9 | 9.75 | 1.58 | 8.18 | 4.25 | 0.3, 11 |
| chime6_eval | 122.7 | 13.92 | 3.57 | 10.35 | 12.54 | 0.6, 11 |
| dev | 329.2 | 7.12 | 1.77 | 5.35 | 5.27 | 0.5, 11 |
| dipco_dev | 98.1 | 11.67 | 2.64 | 9.04 | 7.67 | 0.5, 11 |
| dipco_eval | 92.0 | 9.96 | 3.56 | 6.41 | 8.53 | 0.6, 11 |
| earnings21_eval | 32.8 | 9.50 | 6.94 | 2.56 | 9.50 | 0.8, 11 |
| icsi_dev | 12.5 | 11.03 | 2.83 | 8.20 | 10.35 | 0.7, 11 |
| icsi_eval | 11.9 | 15.16 | 2.25 | 12.91 | 12.67 | 0.5, 11 |
| notsofar_dev | 123.7 | 3.39 | 0.87 | 2.52 | 2.89 | 0.7, 11 |
| notsofar_eval | 398.5 | 3.24 | 1.58 | 1.65 | 3.15 | 0.7, 11 |
| ten_vad_eval | 0.1 | 19.52 | 17.53 | 1.98 | 16.97 | 0.9, 11 |
| voxconverse_dev | 18.9 | 4.54 | 2.58 | 1.96 | 4.48 | 0.7, 11 |
| voxconverse_eval | 39.0 | 7.42 | 5.72 | 1.70 | 7.42 | 0.8, 11 |
Hour-weighted DetER: 7.61. Macro average over the 21 sets: 10.19.
Training
Recipe: egs2/frontia_vad_v1/diar1 in ESPnet.
The config is in this repository. Weights: the average of the two checkpoints with the best validation accuracy, epochs 11 and 17.
| training data | licence |
|---|---|
| AMI (every microphone) | CC BY 4.0 |
| CHiME-6, OpenSLR SLR150 (every array and binaural channel) | CC BY-SA 4.0 (the ShareAlike condition is to be removed shortly, leaving CC BY 4.0) |
| NOTSOFAR-1 240825.1 | CC BY 4.0 |
| ICSI | CC BY 4.0 |
noise: MUSAN noise/ and music/; CHiME-3 background |
CC licences (some music/ tracks CC BY-SA) / public domain; CC BY 4.0 |
The other sets in the table above are used for evaluation only.
Training noise
The training audio was augmented by adding recorded noise to it. That noise
audio is not distributed with this model; it was used only as additive
background noise during training. All of MUSAN music/ was used, and some of
those tracks are licensed CC BY-SA. The next release restricts the music to
MUSAN tracks without vocals that are CC0, CC BY or public domain.
Licence
CC BY-SA 4.0. The training data include material under CC BY-SA: CHiME-6 and
some MUSAN music/ tracks.
- Downloads last month
- 67