README.md

---
license: other
license_name: mixed-provenance
license_link: https://huggingface.co/Scicom-intl/CallEnhancer-Lite#licence
language: [en, ms, zh]
tags: [speech-enhancement, speech-restoration, call-centre, telephony, distilhubert, realtime]
base_model: ntu-spml/distilhubert
pipeline_tag: audio-to-audio
---

# CallEnhancer-Lite — realtime call-centre speech restoration

A realtime-oriented sibling of [`Scicom-intl/CallEnhancer`](https://huggingface.co/Scicom-intl/CallEnhancer)
(enhancer-small) and [`Scicom-intl/CallEnhancer-Base`](https://huggingface.co/Scicom-intl/CallEnhancer-Base)
(enhancer-base). Same two-stage recipe — a feature predictor followed by a DAC vocoder —
but the 580M w2v-BERT 2.0 encoder is replaced by **DistilHuBERT (2 transformer layers)**
plus a small predictor head, and the target degradation is narrowed to three jobs:
**call-centre channel**, **background-noise removal**, and **cross-talk**.

|  | encoder | decoder | total |
|---|---|---|---|
| enhancer-small | w2v-BERT 24L + LoRA — 580M | DAC 3072ch @48 kHz — 188M | 768M |
| enhancer-base | w2v-BERT 24L + LoRA — 580M | DAC 3072ch +extra_res — 381M | 961M |
| **enhancer-lite** | **DistilHuBERT 2L + head — 37M** | DAC 1536ch @24 kHz — 52M | **89M** |

## ⚠️ Status: stage 1 only — this is not yet a usable enhancer

**This repo currently contains the ENCODER only.** It maps degraded 16 kHz audio to
1024-d SSL features; it does **not** produce audio. The matching 24 kHz decoder is still
training and will be added as `decoder_lite/`. Until then these weights are useful for
resuming training or for feature-level work, not for restoring calls.

## What the encoder does

```
degraded 16 kHz --> [DistilHuBERT 2L + PredictorHead] --> features [T, 1024] @ 50 Hz
```

It is distilled against a **frozen w2v-BERT 2.0 (24L) teacher running on the clean
audio**, so it predicts the same 1024-d feature space the existing CallEnhancer decoders
consume. Both stacks emit 50 Hz features (DistilHuBERT's conv frontend strides to 320
samples @16 kHz), which is exactly the DAC decoder's frame rate.

## Results

Feature-space error against the clean teacher, as `nmse = MSE / var(target) = 1 − R²`
(scale-free; 1.0 = no better than predicting the mean, 0.0 = exact):

| model | encoder params | nmse |
|---|---|---|
| enhancer-small FE (w2v-BERT 24L + LoRA) | 580M | ~0.147 |
| **enhancer-lite FE @964k (this checkpoint)** | **37M** | **0.1403** |

The 15× smaller encoder matches the large one on this metric. The LR schedule was the
whole story: constant 1e-4 to 200k (plateau at 0.1652), then cosine decay with SGDR-style
warm restarts — 0.1652 → 0.1505 @300k → 0.1458 @500k → 0.1403 @964k.

Caveats for honest comparison: the two runs use different degradation menus, and ~40% of
the loss is measured on zero-padding in both (median training file is 6.7 s against a
12 s window). The quoted figure is a rolling mean over 10k steps; per-step values swing
by roughly ±0.007 around it.

## Speed

Measured on 16 CPU threads, 2 s chunk, batch 1, fp32:

| stack | encoder | decoder | total | xRT |
|---|---|---|---|---|
| enhancer-small | 137.1 ms | 468.2 ms | 605.2 ms | 3.3 |
| **enhancer-lite** (37M + 52M @24 kHz) | **18.7 ms** | **114.4 ms** | **133.1 ms** | **15.0** |

The encoder is 6.7× faster; the decoder shrink is what delivers the rest. On GPU both
stacks are far past realtime, so the encoder swap's value there is sequential depth
(24 attention blocks → 2), weight traffic (1.2 GB → 74 MB bf16) and no O(T²) attention —
i.e. small-chunk streaming latency and per-stream memory.

Algorithmic lookahead: 80 ms (2 head blocks × kernel 5 at 50 Hz).

## Files

| file | what |
|---|---|
| `fe_lite/fe_only.pt` | encoder weights (~148 MB) — for inference |
| `fe_lite/last.pt` | full train state incl. optimizer (~443 MB) — resumable |

## Usage

```python
import torch, fe_distilhubert as fedh   # runpod/fe_distilhubert.py in the Sidon repo

fe   = fedh.load_fe("fe_lite/fe_only.pt", torch.device("cuda"))
proc = fedh.make_processor()
batch = fedh.collate_wavs([wav_16k_float32], proc)      # raw waveform in
feats = fe(**{k: v.cuda() for k, v in batch.items()}).last_hidden_state   # [1, T, 1024]
```

No `peft` needed — unlike the LoRA-based enhancer-small FE, this is a full finetune.

## Training

* **Degradation** (all applied online, never precomputed): telephony channel 85%
  (200–350 Hz high-pass → random 2.8–4.2 kHz ceiling via an 8/11.025/12/16 kHz bottleneck
  → GSM / G.711 µ-law / none → 16–40 kbps MP3 → line noise → VoIP dropouts); caller-side
  background noise 85% at 0–22 dB SNR from a real corpus; **cross-talk 50%** (1–2
  interferer speakers, each active in a random 30–80% sub-segment with raised-cosine
  fades, optionally passed through their own phone channel, SIR 2–15 dB, plus half-duplex
  ducking); babble 20%. Reverb, echo, AGC, hum, codec tandeming, packet-loss concealment
  and handset EQ are deliberately **off** — the small model spends its capacity on three
  problems rather than ten.
* Cross-talk and babble are mixed into the **input only**; the target remains the clean
  dominant speaker, so "restore the loudest talker" is well defined.
* 964k steps, effective batch 32, 12 s windows, bf16, 2× H20.
* Teacher pool: ~2,700 h of ≥44.1 kHz speech, DNSMOS-filtered, spanning read, audiobook,
  podcast and conversational registers across many languages.

## Licence

`other` — provenance is mixed and not uniformly permissive. The base model
(`ntu-spml/distilhubert`) is Apache-2.0, but the noise corpus used during training
includes CC BY-NC and share-alike material, and parts of the speech pool carry
non-commercial terms. Review the upstream sources before any commercial use.