|
|
--- |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# Spoken-Dialogue-Turn-Detection |
|
|
|
|
|
[](https://github.com/Akshay090/svg-banners) |
|
|
|
|
|
[](https://github.com/wbb921/speech-turn-detection/stargazers) |
|
|
[](https://huggingface.co/luht/speech-turn-detection/tree/main) |
|
|
 |
|
|
[](https://github.com/wbb921/speech-turn-detection/discussions) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Spoken Dialogue Turn Detection refers to distinguishing between a short pause and the actual end of a user’s query. |
|
|
|
|
|
Traditional approaches rely on Voice Activity Detection (VAD) with a fixed delay, which often misinterprets short pauses as endpoints, leading to delayed responses or premature cut-offs. |
|
|
|
|
|
This repository provides an implementation of spoken dialogue turn detection, which directly takes speech as inputs instead of texts and outputs turn-taking patterns along with speaker turns. |
|
|
|
|
|
|
|
|
## Installation |
|
|
|
|
|
```bash |
|
|
conda create -n turn-detection python=3.10 |
|
|
apt-get install libsndfile1 |
|
|
git clone https://github.com/wbb921/spoken-dialogue-turn-detection.git |
|
|
cd spoken-dialogue-turn-detection |
|
|
pip install -r requirements |
|
|
``` |
|
|
|
|
|
## Checkpoints |
|
|
|
|
|
The model is trained on SpokenWOZ (249h) and Fisher (1960h) |
|
|
|
|
|
The checkpoints can be downloaded from |
|
|
|
|
|
https://huggingface.co/luht/speech-turn-detection/blob/main/model_spokenwoz.pt |
|
|
|
|
|
https://huggingface.co/luht/speech-turn-detection/blob/main/model_fisher_spokenwoz.pt |
|
|
|
|
|
place the pt file under the ckpt directory once downloaded |
|
|
|
|
|
## Model Inputs/Outputs |
|
|
|
|
|
### Inputs |
|
|
|
|
|
Inputs should be stereo audios, with 24kHz sampling rate, some samples can be found in the "data/" directory |
|
|
|
|
|
### Outputs |
|
|
|
|
|
The model outputs several turn-taking patterns: IPU(0), Listen(1), Gap(2), Pause(3), Overlap(4). Gap refers to mutual silence with speaker change before and after. Pause refers to mutual silence without speaker change. |
|
|
|
|
|
The endpoint(speaker turn point) can be seen as the timestamp where IPU(0) turns into Gap(2). |
|
|
|
|
|
|
|
|
The outputs will be |
|
|
```bash |
|
|
## Channel 0 State Transitions ## |
|
|
0.00s -> 2.88s ( 2.88s) | State: Gap |
|
|
2.88s -> 3.28s ( 0.40s) | State: Speak |
|
|
3.28s -> 4.08s ( 0.80s) | State: Gap |
|
|
...... |
|
|
|
|
|
## Channel 1 State Transitions ## |
|
|
0.00s -> 2.88s ( 2.88s) | State: Gap |
|
|
2.88s -> 3.28s ( 0.40s) | State: Listen |
|
|
3.28s -> 4.08s ( 0.80s) | State: Gap |
|
|
``` |
|
|
which is printed on the screen |
|
|
|
|
|
and a numpy array which stores the turn-taking patterns as defined above with shape (2, T) |
|
|
|
|
|
The model outputs with a frequency of 12.5Hz (80 ms a frame) |
|
|
|
|
|
## Usage |
|
|
|
|
|
The model is totally causal, which can be used in offline or streaming manner. |
|
|
|
|
|
Offline inference |
|
|
```bash |
|
|
python infer.py --audio_path "./data/MUL0001.wav" --checkpoint_path "./ckpt/model_spokenwoz.pt" --output_dir "./inference_results" |
|
|
``` |
|
|
|
|
|
Streaming Inference |
|
|
```bash |
|
|
python infer_streaming.py --audio_path "./data/MUL0001.wav" --checkpoint_path "./ckpt/model_spokenwoz.pt" --output_dir "./inference_results" |
|
|
``` |
|
|
|
|
|
The turn-taking states will be printed on the screen, while the numpy array which stores the turn-taking patterns will be saved in ./inference_results with the same name as the input audio, e.g. "MUL0001.npy" |
|
|
|
|
|
## Train |
|
|
|
|
|
### Data Preparation |
|
|
|
|
|
Two things have to be prepared for training: |
|
|
|
|
|
1. Training audio files (24kHz, 16-bit, stereo), placed under /path/to/your/audio_dir: |
|
|
```bash |
|
|
audio_1.wav |
|
|
audio_2.wav |
|
|
audio_3.wav |
|
|
... |
|
|
``` |
|
|
2. Turn-taking pattern labels, numpy arrays, same name as the training audio files, placed under /path/to/your/label_dir: |
|
|
```bash |
|
|
audio_1.npy |
|
|
audio_2.npy |
|
|
audio_3.npy |
|
|
... |
|
|
``` |
|
|
Turn-taking pattern labels' time frequency is 12.5 Hz (80 ms a frame), the shape of the numpy array should be (2, T), T = audio_duration / 80ms. |
|
|
|
|
|
In the 'data_utils' directory, you can find scripts for preparing turn-taking pattern labels from SpokenWOZ dataset annotations: |
|
|
|
|
|
1. Using silero_vad to refine the utterance timestamps. |
|
|
2. Generating the turn-taking labels. |
|
|
|
|
|
### Start Training |
|
|
|
|
|
After data preparation, use the following command to start training: |
|
|
|
|
|
```bash |
|
|
python train.py --audio_dir /path/to/your/audio_dir --label_dir /path/to/your/label_dir --batch_size 32 --exp_name test |
|
|
``` |
|
|
## Results |
|
|
|
|
|
The model achieves an ep-cutoff rate of 4.72% on SpokenWOZ test set. |
|
|
| Method | ep-50 (ms) | ep-90 (ms) | ep-cutoff (%) | |
|
|
|------------------------------|------------|------------|---------------| |
|
|
| Silero_vad (200ms latency) | 240 | 320 | 35.86 | |
|
|
| Silero_vad (500ms latency) | 560 | 640 | 23.11 | |
|
|
| The proposed model | 80 | 400 | 4.72 | |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
