license: other
license_name: mixed-provenance
license_link: https://huggingface.co/Scicom-intl/CallEnhancer-Lite#licence
language:
- en
- ms
- zh
tags:
- speech-enhancement
- speech-restoration
- call-centre
- telephony
- distilhubert
- realtime
base_model: ntu-spml/distilhubert
pipeline_tag: audio-to-audio
CallEnhancer-Lite — realtime call-centre speech restoration
A realtime-oriented sibling of Scicom-intl/CallEnhancer
(enhancer-small) and Scicom-intl/CallEnhancer-Base
(enhancer-base). Same two-stage recipe — a feature predictor followed by a DAC vocoder —
but the 580M w2v-BERT 2.0 encoder is replaced by DistilHuBERT (2 transformer layers)
plus a small predictor head, and the target degradation is narrowed to three jobs:
call-centre channel, background-noise removal, and cross-talk.
| encoder | decoder | total | |
|---|---|---|---|
| enhancer-small | w2v-BERT 24L + LoRA — 580M | DAC 3072ch @48 kHz — 188M | 768M |
| enhancer-base | w2v-BERT 24L + LoRA — 580M | DAC 3072ch +extra_res — 381M | 961M |
| enhancer-lite | DistilHuBERT 2L + head — 37M | DAC 1536ch @24 kHz — 52M | 89M |
⚠️ Status: stage 1 only — this is not yet a usable enhancer
This repo currently contains the ENCODER only. It maps degraded 16 kHz audio to
1024-d SSL features; it does not produce audio. The matching 24 kHz decoder is still
training and will be added as decoder_lite/. Until then these weights are useful for
resuming training or for feature-level work, not for restoring calls.
What the encoder does
degraded 16 kHz --> [DistilHuBERT 2L + PredictorHead] --> features [T, 1024] @ 50 Hz
It is distilled against a frozen w2v-BERT 2.0 (24L) teacher running on the clean audio, so it predicts the same 1024-d feature space the existing CallEnhancer decoders consume. Both stacks emit 50 Hz features (DistilHuBERT's conv frontend strides to 320 samples @16 kHz), which is exactly the DAC decoder's frame rate.
Results
Feature-space error against the clean teacher, as nmse = MSE / var(target) = 1 − R²
(scale-free; 1.0 = no better than predicting the mean, 0.0 = exact):
| model | encoder params | nmse |
|---|---|---|
| enhancer-small FE (w2v-BERT 24L + LoRA) | 580M | ~0.147 |
| enhancer-lite FE @500k (this checkpoint) | 37M | 0.1458 |
The 15× smaller encoder matches the large one on this metric. Training used a constant LR to 200k (plateau at 0.1652), then cosine decay with SGDR-style warm restarts: 0.1652 → 0.1505 @300k → 0.1458 @500k.
Caveats for honest comparison: the two runs use different degradation menus, and ~40% of the loss is measured on zero-padding in both (median training file is 6.7 s against a 12 s window).
Speed
Measured on 16 CPU threads, 2 s chunk, batch 1, fp32:
| stack | encoder | decoder | total | xRT |
|---|---|---|---|---|
| enhancer-small | 137.1 ms | 468.2 ms | 605.2 ms | 3.3 |
| enhancer-lite (37M + 52M @24 kHz) | 18.7 ms | 114.4 ms | 133.1 ms | 15.0 |
The encoder is 6.7× faster; the decoder shrink is what delivers the rest. On GPU both stacks are far past realtime, so the encoder swap's value there is sequential depth (24 attention blocks → 2), weight traffic (1.2 GB → 74 MB bf16) and no O(T²) attention — i.e. small-chunk streaming latency and per-stream memory.
Algorithmic lookahead: 80 ms (2 head blocks × kernel 5 at 50 Hz).
Files
| file | what |
|---|---|
fe_lite/fe_only.pt |
encoder weights (~148 MB) — for inference |
fe_lite/last.pt |
full train state incl. optimizer (~443 MB) — resumable |
Usage
import torch, fe_distilhubert as fedh # runpod/fe_distilhubert.py in the Sidon repo
fe = fedh.load_fe("fe_lite/fe_only.pt", torch.device("cuda"))
proc = fedh.make_processor()
batch = fedh.collate_wavs([wav_16k_float32], proc) # raw waveform in
feats = fe(**{k: v.cuda() for k, v in batch.items()}).last_hidden_state # [1, T, 1024]
No peft needed — unlike the LoRA-based enhancer-small FE, this is a full finetune.
Training
- Degradation (all applied online, never precomputed): telephony channel 85% (200–350 Hz high-pass → random 2.8–4.2 kHz ceiling via an 8/11.025/12/16 kHz bottleneck → GSM / G.711 µ-law / none → 16–40 kbps MP3 → line noise → VoIP dropouts); caller-side background noise 85% at 0–22 dB SNR from a real corpus; cross-talk 50% (1–2 interferer speakers, each active in a random 30–80% sub-segment with raised-cosine fades, optionally passed through their own phone channel, SIR 2–15 dB, plus half-duplex ducking); babble 20%. Reverb, echo, AGC, hum, codec tandeming, packet-loss concealment and handset EQ are deliberately off — the small model spends its capacity on three problems rather than ten.
- Cross-talk and babble are mixed into the input only; the target remains the clean dominant speaker, so "restore the loudest talker" is well defined.
- 500k steps, effective batch 32, 12 s windows, bf16, 2× H20.
- Teacher pool: ~2,700 h of ≥44.1 kHz speech, DNSMOS-filtered, spanning read, audiobook, podcast and conversational registers across many languages.
Licence
other — provenance is mixed and not uniformly permissive. The base model
(ntu-spml/distilhubert) is Apache-2.0, but the noise corpus used during training
includes CC BY-NC and share-alike material, and parts of the speech pool carry
non-commercial terms. Review the upstream sources before any commercial use.