DialogueSidon2

Two-speaker dialogue separation and restoration. Given a degraded recording of a two-person conversation (telephone channels, noise, reverberation, codecs), DialogueSidon2 returns one clean 24 kHz track per speaker. It is the successor of DialogueSidon.

Demo: sarulab-speech/DialogueSidon2-demo

Model

File	Description
model.safetensors, config.json	The separation model: w2v-BERT 2.0 (13 layers, fully fine-tuned), the readout head and the SSL-VAE
ssl_vae.ckpt, latent_statistics.pt	The stage 1 SSL-VAE and its latent statistics, for training a separation model on this latent space
separator.pt2, vae_decoder.pt2, metadata.json	torch.export programs of the separation model, used by the demo
ds2_infer.py	Standalone inference with the exported programs (needs only torch and torchaudio)

Unlike DialogueSidon, which samples the latents with a diffusion model, DialogueSidon2 reads each speaker's latents directly from the mixture in a single forward pass. There is no sampling: the same input always gives the same output. The SSL-VAE has DialogueSidon's architecture, retrained on single-speaker speech restoration data.

The model was trained on 30 s segments for 200,000 steps. Inputs up to 30 s are separated in one shot; longer inputs are separated in 30 s windows with 5 s of overlap, re-aligned across windows by the correlation of the outputs in the overlap, and cross-faded.

The training and inference code is in sarulab-speech/Sidon (src/sidon/dialoguesidon2/, src/sidon/model/dialoguesidon2/), with the recipes that produced the training data.

Usage

With the Sidon repository:

python -m sidon.dialoguesidon2.infer --model <download of this repository> --output-dir out dialogue.wav
# out/dialogue_speaker1.wav and out/dialogue_speaker2.wav (24 kHz)

Without it, from the exported programs:

import sys
import soundfile as sf
import torch
from huggingface_hub import snapshot_download

model_dir = snapshot_download("sarulab-speech/DialogueSidon2",
                              allow_patterns=["separator.pt2", "vae_decoder.pt2", "metadata.json", "ds2_infer.py"])
sys.path.insert(0, model_dir)
import ds2_infer

models = ds2_infer.load(model_dir, "cuda")  # or "cpu"
audio, rate = sf.read("dialogue.wav", always_2d=True)  # mono or stereo, any sample rate
speakers = ds2_infer.separate(torch.from_numpy(audio.T).float(), rate, models)  # [2, N] at 24 kHz
sf.write("speakers.wav", speakers.T.numpy(), ds2_infer.SAMPLE_RATE)  # left: speaker 1, right: speaker 2

The programs were exported with torch.export from PyTorch 2.8.0; load them with torch==2.8.0 and torchaudio==2.8.0.

Evaluation

The evaluation sets and metrics of the DialogueSidon paper: WER with Whisper large-v3 (p-CER on CallFriend), NISQA, DNSMOS, speaker similarity (WavLM) and VAD accuracy. Both models were scored in the same run on the same items. DialogueSidon is the released model (30 sampling steps). On CallFriend, the output tracks are assigned to the speakers by character errors.

Switchboard (SWB) evaluation set, 100 dialogues of 20 s. †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.

Method	WER (%) ↓	NISQA ↑	DNSMOS ↑	Spk. Sim. ↑	VAD Acc. ↑
	Content	Recording Quality	Recording Quality	Speaker	Separation
Noisy†	60.812	2.315	3.005	—	—
DialogueSidon	14.432	3.453	3.641	0.887	0.938
DialogueSidon2	12.706	3.634	3.756	0.895	0.942

CallFriend, 20 dialogues of 20 s per language. †Noisy does not perform speech separation. Bold: the better of DialogueSidon and DialogueSidon2.

Language	Method	p-CER (%) ↓	NISQA ↑	DNSMOS ↑	Spk. Sim. ↑	VAD Acc. ↑
		Content	Recording Quality	Recording Quality	Speaker	Separation
German	Noisy†	—	2.131	3.001	—	—
	DialogueSidon	14.225	3.203	3.541	0.902	0.954
	DialogueSidon2	12.860	3.602	3.675	0.895	0.962
French	Noisy†	—	2.353	3.178	—	—
	DialogueSidon	22.683	3.393	3.639	0.906	0.948
	DialogueSidon2	23.895	3.654	3.786	0.907	0.941
Japanese	Noisy†	—	2.177	2.897	—	—
	DialogueSidon	19.911	3.157	3.335	0.929	0.934
	DialogueSidon2	21.411	3.530	3.486	0.941	0.940
Spanish	Noisy†	—	2.235	2.919	—	—
	DialogueSidon	18.354	3.155	3.324	0.865	0.933
	DialogueSidon2	19.375	3.361	3.533	0.868	0.941
Mandarin	Noisy†	—	2.380	3.214	—	—
	DialogueSidon	30.565	3.432	3.532	0.925	0.916
	DialogueSidon2	25.471	3.803	3.805	0.942	0.921

OpenDialog (in-the-wild data), 100 recordings under 30 s. †Noisy does not perform speech separation; its metrics are computed on the monaural mixture. Bold: the better of DialogueSidon and DialogueSidon2.

Method	WER (%) ↓	NISQA ↑	DNSMOS ↑
	Content	Recording Quality	Recording Quality
Noisy†	—	3.826	3.615
DialogueSidon	14.073	3.568	3.598
DialogueSidon2	9.955	3.819	3.713

Training data

Separation model (two-speaker mixtures, 30 s segments):

• The DialogueSidon training data: conversational telephone speech (CallHome, Fisher).
• OpenYap-1k (CC BY 4.0).
• Basis Conversations 1500, published by Basis: basis-ai/basis-conversations-1500, revision 05a19e06b5eed44d701a8860b7e2336a44f762dd.
• Simulated two-speaker mixtures of LibriTTS-R (CC BY 4.0).

SSL-VAE (single-speaker speech): LibriTTS-R, FLEURS-R, JVS, Hi-Fi-CAPTAIN, BibleTTS, EARS, Expresso, JSUT and VCTK.

Initialization: facebook/w2v-bert-2.0 (MIT).

Intended use and limitations

• Research and non-commercial use (CC BY-NC 4.0).
• Exactly two speakers. The order of the two output tracks is arbitrary.
• Long inputs are processed in windows. If a speaker is silent throughout an overlap, the re-alignment can swap the two tracks from that window on.
• Do not use the model for voice cloning, for speaker identification, or on recordings of people who have not consented to processing.

License

CC BY-NC 4.0