CallEnhancer-Lite — realtime call-centre speech restoration

A realtime-oriented sibling of Scicom-intl/CallEnhancer (enhancer-small) and Scicom-intl/CallEnhancer-Base (enhancer-base). Same two-stage recipe — a feature predictor followed by a DAC vocoder — but the 580M w2v-BERT 2.0 encoder is replaced by DistilHuBERT (2 transformer layers) plus a small predictor head, and the target degradation is narrowed to three jobs: call-centre channel, background-noise removal, and cross-talk.

	encoder	decoder	total
enhancer-small	w2v-BERT 24L + LoRA — 580M	DAC 3072ch @48 kHz — 188M	768M
enhancer-base	w2v-BERT 24L + LoRA — 580M	DAC 3072ch +extra_res — 381M	961M
enhancer-lite	DistilHuBERT 2L + head — 37M	DAC 1536ch @24 kHz — 52M	89M

⚠️ Status: stage 1 only — this is not yet a usable enhancer

This repo currently contains the ENCODER only. It maps degraded 16 kHz audio to 1024-d SSL features; it does not produce audio. The matching 24 kHz decoder is still training and will be added as decoder_lite/. Until then these weights are useful for resuming training or for feature-level work, not for restoring calls.

What the encoder does

degraded 16 kHz --> [DistilHuBERT 2L + PredictorHead] --> features [T, 1024] @ 50 Hz

It is distilled against a frozen w2v-BERT 2.0 (24L) teacher running on the clean audio, so it predicts the same 1024-d feature space the existing CallEnhancer decoders consume. Both stacks emit 50 Hz features (DistilHuBERT's conv frontend strides to 320 samples @16 kHz), which is exactly the DAC decoder's frame rate.

Results

Feature-space error against the clean teacher, as nmse = MSE / var(target) = 1 − R² (scale-free; 1.0 = no better than predicting the mean, 0.0 = exact):

model	encoder params	nmse
enhancer-small FE (w2v-BERT 24L + LoRA)	580M	~0.147
enhancer-lite FE @500k (this checkpoint)	37M	0.1458

The 15× smaller encoder matches the large one on this metric. Training used a constant LR to 200k (plateau at 0.1652), then cosine decay with SGDR-style warm restarts: 0.1652 → 0.1505 @300k → 0.1458 @500k.

Caveats for honest comparison: the two runs use different degradation menus, and ~40% of the loss is measured on zero-padding in both (median training file is 6.7 s against a 12 s window).

Speed

Measured on 16 CPU threads, 2 s chunk, batch 1, fp32:

stack	encoder	decoder	total	xRT
enhancer-small	137.1 ms	468.2 ms	605.2 ms	3.3
enhancer-lite (37M + 52M @24 kHz)	18.7 ms	114.4 ms	133.1 ms	15.0

The encoder is 6.7× faster; the decoder shrink is what delivers the rest. On GPU both stacks are far past realtime, so the encoder swap's value there is sequential depth (24 attention blocks → 2), weight traffic (1.2 GB → 74 MB bf16) and no O(T²) attention — i.e. small-chunk streaming latency and per-stream memory.

Algorithmic lookahead: 80 ms (2 head blocks × kernel 5 at 50 Hz).

Files

file	what
fe_lite/fe_only.pt	encoder weights (~148 MB) — for inference
fe_lite/last.pt	full train state incl. optimizer (~443 MB) — resumable

Usage

import torch, fe_distilhubert as fedh   # runpod/fe_distilhubert.py in the Sidon repo

fe   = fedh.load_fe("fe_lite/fe_only.pt", torch.device("cuda"))
proc = fedh.make_processor()
batch = fedh.collate_wavs([wav_16k_float32], proc)      # raw waveform in
feats = fe(**{k: v.cuda() for k, v in batch.items()}).last_hidden_state   # [1, T, 1024]

No peft needed — unlike the LoRA-based enhancer-small FE, this is a full finetune.

Training

• Degradation (all applied online, never precomputed): telephony channel 85% (200–350 Hz high-pass → random 2.8–4.2 kHz ceiling via an 8/11.025/12/16 kHz bottleneck → GSM / G.711 µ-law / none → 16–40 kbps MP3 → line noise → VoIP dropouts); caller-side background noise 85% at 0–22 dB SNR from a real corpus; cross-talk 50% (1–2 interferer speakers, each active in a random 30–80% sub-segment with raised-cosine fades, optionally passed through their own phone channel, SIR 2–15 dB, plus half-duplex ducking); babble 20%. Reverb, echo, AGC, hum, codec tandeming, packet-loss concealment and handset EQ are deliberately off — the small model spends its capacity on three problems rather than ten.
• Cross-talk and babble are mixed into the input only; the target remains the clean dominant speaker, so "restore the loudest talker" is well defined.
• 500k steps, effective batch 32, 12 s windows, bf16, 2× H20.
• Teacher pool: ~2,700 h of ≥44.1 kHz speech, DNSMOS-filtered, spanning read, audiobook, podcast and conversational registers across many languages.

Licence

other — provenance is mixed and not uniformly permissive. The base model (ntu-spml/distilhubert) is Apache-2.0, but the noise corpus used during training includes CC BY-NC and share-alike material, and parts of the speech pool carry non-commercial terms. Review the upstream sources before any commercial use.