| --- |
| tags: |
| - speech-separation |
| - audio |
| - dprnn |
| - multi-decoder |
| license: mit |
| base_model: JunzheJosephZhu/MultiDecoderDPRNN |
| datasets: |
| - custom |
| language: |
| - en |
| --- |
| |
| # Multi-Decoder DPRNN — Fine-Tuned for 1–5 Speaker Separation |
|
|
| Fine-tuned version of [MultiDecoderDPRNN](https://huggingface.co/JunzheJosephZhu/MultiDecoderDPRNN) for variable speaker count (1–5) speech separation. |
|
|
| ## What changed? |
|
|
| The original pre-trained model supports 2–5 speakers and always outputs 5 active sources regardless of actual speaker count (~22% speaker count accuracy). Our fine-tuning teaches the model to: |
| - Correctly identify the number of speakers (88% accuracy) |
| - Output silence on unused channels |
| - Support single-speaker scenarios (new 1-speaker decoder) |
|
|
| ## Checkpoints |
|
|
| | Checkpoint | Architecture | SI-SDR | SI-SDRi | SDR | Spk Acc | File | |
| |------------|-------------|--------|---------|-----|---------|------| |
| | **Rys (best)** | n_srcs=[1,2,3,4,5] | +7.33 dB | +10.01 dB | +7.32 dB | 88.0% | `weights/dprnn_rys_weights.pt` | |
| | Zhaksh | n_srcs=[2,3,4,5] | +3.24 dB | +5.92 dB | +3.24 dB | 50.0% | `weights/dprnn_zhaksh_weights.pt` | |
| | Original (baseline) | n_srcs=[2,3,4,5] | -1.29 dB | +1.39 dB | -1.29 dB | 22.0% | — | |
| |
| ### SI-SDRi by Speaker Count |
| |
| | N | Original | Rys | Zhaksh | |
| |---|----------|-----|--------| |
| | 1 | -9.17 dB | **+8.86 dB** | -7.47 dB | |
| | 2 | +4.07 dB | **+11.98 dB** | +11.58 dB | |
| | 3 | +2.46 dB | **+10.00 dB** | +9.78 dB | |
| | 4 | +4.56 dB | **+9.65 dB** | +7.78 dB | |
| | 5 | +5.05 dB | **+9.56 dB** | +7.94 dB | |
| |
| ## Usage |
| |
| ```python |
| import sys, torch |
| sys.path.insert(0, "asteroid/egs/wsj0-mix-var/Multi-Decoder-DPRNN") |
| from model import MultiDecoderDPRNN |
| from huggingface_hub import hf_hub_download |
|
|
| # Load base architecture with extended speaker count |
| pretrained = MultiDecoderDPRNN.from_pretrained("JunzheJosephZhu/MultiDecoderDPRNN") |
| fb = pretrained.encoder.filterbank |
| cfg = dict( |
| bn_chan=pretrained.masker.bn_chan, hid_size=pretrained.masker.hid_size, |
| chunk_size=pretrained.masker.chunk_size, hop_size=pretrained.masker.hop_size, |
| n_repeats=pretrained.masker.n_repeats, norm_type=pretrained.masker.norm_type, |
| bidirectional=pretrained.masker.bidirectional, rnn_type=pretrained.masker.rnn_type, |
| num_layers=pretrained.masker.num_layers, dropout=pretrained.masker.dropout, |
| n_filters=fb.n_feats_out, kernel_size=fb.kernel_size, |
| stride=fb.stride, sample_rate=8000, |
| ) |
| |
| model = MultiDecoderDPRNN(n_srcs=[1, 2, 3, 4, 5], **cfg) |
| weights = hf_hub_download("Namadgi/MultiDecoderDPRNN-finetuned", "weights/dprnn_rys_weights.pt") |
| model.load_state_dict(torch.load(weights, map_location="cpu")) |
| model.eval() |
| |
| # Inference |
| mix = torch.randn(1, 32000) # [batch, time] at 8kHz |
| reconstructed, selector = model(mix, ground_truth=[3]) # 3 speakers |
| # reconstructed: [1, n_stages, max_spks, T] |
| # selector: [1, n_stages, n_decoders] |
| ``` |
| |
| ## Training Details |
| |
| - **Base model:** JunzheJosephZhu/MultiDecoderDPRNN |
| - **Dataset:** 10,000 synthetic mixtures (LibriSpeech train-clean-100 + WHAM! noise), 1–5 speakers |
| - **Evaluation:** 50 samples from held-out CV set (LibriSpeech dev-clean + WHAM! cv) |
| - **Sample rate:** 8 kHz |
| - **Loss:** Multi-stage PIT SI-SDR + Cross-entropy selector loss |
| |
| ## Authors |
| |
| - Rys ([@RysNamadgi](https://huggingface.co/RysNamadgi)) |
| - Zhaksh |
| |
| Part of the [Namadgi](https://huggingface.co/Namadgi) research group. |
| |
