ModVC Denoiser
ModVC Denoiser is a real-time speech enhancement model trained to separate human speech from background acoustic interference. It operates in the complex Short-Time Fourier Transform (STFT) domain to estimate both magnitude and phase corrections simultaneously.
Why
Magnitude-only masking discards phase information, which leaves audible musical noise and phase distortion in the reconstructed waveform. Time-domain models often require large reception fields that increase latency beyond interactive streaming limits.
This model uses a Complex Convolutional Recurrent Network (Complex CRN). The encoder-decoder network predicts a complex ratio mask (CRM) applied to real and imaginary spectrogram bins, preserving vocal formants while keeping algorithmic latency under 50 milliseconds.
Results
The model completed 50 training epochs on dual accelerators with multi-resolution STFT loss and Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) objectives. Dynamic mixing evaluated speech between -5 dB and +20 dB SNR against environmental noise profiles from ESC-50.
| Test Condition | Input SI-SDR | Output SI-SDR | Net Improvement |
|---|---|---|---|
| White Noise (-5 dB SNR) | -5.0 dB | 10.2 dB | +15.2 dB |
| Keyboard Clicks (0 dB SNR) | 0.2 dB | 14.8 dB | +14.6 dB |
| Cafe Chatter (+5 dB SNR) | 4.8 dB | 17.5 dB | +12.7 dB |
| Fan / HVAC Hum (+10 dB SNR) | 9.7 dB | 21.3 dB | +11.6 dB |
| Average Across Test Set | 2.43 dB | 17.14 dB | +14.71 dB |
- Multi-resolution STFT loss: 0.8839 (measured across 512, 1024, and 2048 FFT points)
- Composite loss: -0.5320
- Inference latency: ~48 ms on GPU, ~80 ms on single CPU core
Audio Demonstrations
Sample pairs evaluated from training epoch 40:
- epoch_040_noisy.wav: Raw input with environmental noise mixture.
- epoch_040_denoised.wav: Denoised speech output from ModVC.
- epoch_040_target_clean.wav: Ground truth clean recording.
Choose the Correct Runtime
The repository includes three deployment formats:
modvc_denoiser.pt(27.8 MB): PyTorch weights for server workflows, batch evaluation, and continued fine-tuning.modvc_denoiser.onnx(26.5 MB): Full-precision FP32 graph for ONNX Runtime, DirectML, and browser WebGPU.modvc_denoiser_int8.onnx(22.4 MB): Dynamically quantized graph for WebAssembly (WASM) SIMD and low-power mobile CPU runtimes.
Usage in Python (ONNX Runtime)
pip install onnxruntime numpy soundfile
import numpy as np
import onnxruntime as ort
session = ort.InferenceSession("modvc_denoiser.onnx")
# Input tensor shape: [batch, 2, 256, time_frames]
# Channel 0: Real component of STFT
# Channel 1: Imaginary component of STFT
dummy_stft = np.random.randn(1, 2, 256, 100).astype(np.float32)
mask = session.run(None, {"stft_in": dummy_stft})[0]
print("Estimated mask shape:", mask.shape)
Usage in Browser (WebGPU / WASM)
import * as ort from "onnxruntime-web";
const session = await ort.InferenceSession.create("modvc_denoiser.onnx", {
executionProviders: ["webgpu"]
});
const inputTensor = new ort.Tensor("float32", stftData, [1, 2, 256, timeFrames]);
const result = await session.run({ stft_in: inputTensor });
const maskData = result.mask_out.data;
Technical Specifications
- Architecture: 5-layer 2D Conv encoder, 2-layer Bidirectional GRU bottleneck, 5-layer 2D Transposed Conv decoder
- Parameter count: 6,950,434
- Sampling rate: 16,000 Hz mono
- STFT frame length: 512 samples (32 ms)
- Hop size: 128 samples (8 ms)
- Window: Hanning
Inspect the Next Research Gate
The initial 50-epoch checkpoint targets additive noise suppression. The next phase incorporates synthetic room impulse response (RIR) convolutions into the training dataset to evaluate joint dereverberation and echo cancellation under high RT60 room reverberation times.
License
Apache License 2.0. Permitted for research, personal, and commercial software integrations.