| --- |
| license: mit |
| library_name: candle |
| tags: |
| - voice-conversion |
| - vocoder |
| - real-time |
| - low-latency |
| - rust |
| - candle |
| - asmr |
| language: |
| - ja |
| pipeline_tag: audio-to-audio |
| --- |
| |
| # LightVC — model suite |
|
|
| Own-weights model suite for real-time Japanese voice conversion (ASMR / バ美声). |
| Inference is **pure Rust / [Candle](https://github.com/huggingface/candle)** (no Python runtime), |
| target **E2E < 50 ms, CPU, causal**. Code: <https://github.com/kjranyone/LightVC>. License **MIT**. |
|
|
| All weights are **100% own** (no distillation from other voice-conversion models; targets are real audio only). |
|
|
| ## Components |
|
|
| | component | folder | status | notes | |
| |---|---|---|---| |
| | **vocoder (freeC)** | `vocoder/` | ✅ released | low-latency neural vocoder, ear-parity with BigVGAN | |
| | content encoder (G-enc) | `content-encoder/` | ⬜ planned | causal HuBERT-distilled, leakage-suppressed | |
| | VC generator (AdaIN + CIPT) | `vc/` | ⬜ planned | mel generator, multi-reference moe-factor compositor | |
| | prosody / s_art | `prosody/` | ⬜ planned | articulation clone, prosody policy | |
| | voices | `voices/` | ⬜ planned | target / reference embeddings for the compositor | |
| |
| Download the whole suite or one component: |
| |
| ```bash |
| huggingface-cli download mus8tte/lightvc --local-dir models/ # all |
| huggingface-cli download mus8tte/lightvc --include 'vocoder/*' --local-dir models/ # vocoder only |
| ``` |
| |
| --- |
| |
| ## vocoder/ — freeC (low-latency neural vocoder) |
| |
| F0-free ISTFT-head neural vocoder (Vocos-style ConvNeXt backbone + complex-STFT head), 27.9 M params, groups=1. |
| |
| - **Quality**: ear-parity with BigVGAN on the target voice (human-ear gate). |
| - **Latency**: causal; synthesis window 5.8 ms; streaming E2E ≈ 39 ms (K=4) incl. mel-analysis lookahead (< 50 ms budget). |
| - **Realtime**: chunked streaming RTF 0.94 @ K=4, single-thread CPU. |
| - **Parity (Rust/Candle vs PyTorch)**: vocoder mel→wave **SNR 88.5 dB**; end-to-end audio→mel→wave resynth **62 dB**. |
| |
| **Files** |
| |
| | file | what | |
| |---|---| |
| | `vocoder/freeC.safetensors` | weights (fp32). Keys: `embed.*`, `blocks.{0..7}.{dw,norm,pw1,pw2}.*`, `norm.*`, `head.*`. | |
| | `vocoder/mel_basis_44k_2048_128.safetensors` | librosa slaney mel filterbank `[128,1025]` (key `mel_basis`) for the input mel extractor. | |
|
|
| **Grid / config** |
|
|
| - Synthesis: causal, `n_fft = win = 256`, `hop = 128`, `NB = 129`. |
| - Input mel (BigVGAN `mel_spectrogram`): `n_fft = win = 2048`, `hop = 128`, `n_mels = 128`, `sr = 44100`, `fmin = 0`, `fmax = 22050`, Hann, `center=False` + reflect-pad `(n_fft-hop)//2=960`, magnitude `sqrt(·+1e-9)`, `log(clamp(·,1e-5))`. Mel hop = synthesis hop. |
|
|
| **Usage** |
|
|
| ```bash |
| git clone https://github.com/kjranyone/LightVC && cd LightVC |
| huggingface-cli download mus8tte/lightvc --include 'vocoder/*' --local-dir models_dl/ |
| cargo run -p lightvc-app --release -- resynth \ |
| --input in.wav --weights models_dl/vocoder/freeC.safetensors \ |
| --mel-basis models_dl/vocoder/mel_basis_44k_2048_128.safetensors --output out.wav --k 4 |
| ``` |
|
|
| > Note: freeC's "5.8 ms" is the synthesis-window figure; matching-quality streaming needs ~23 ms mel-analysis lookahead (mel trained quasi-centered). A causal-mel retrain restores true ~5.8 ms — see the repo roadmap. |
|
|
| **Training**: female multi-speaker corpus, universal vocoder, full-corpus rolling + AMP. No VC-teacher distillation. |
|
|
