| |
|
| |
|
| | --- |
| | license: apache-2.0 |
| | tags: |
| | - audio |
| | - speech |
| | - audio-to-audio |
| | - speech-language-models |
| | datasets: |
| | - amphion/Emilia-Dataset |
| | - facebook/multilingual_librispeech |
| | - CSTR-Edinburgh/vctk |
| | - google/fleurs |
| | - mozilla-foundation/common_voice_13_0 |
| | - mythicinfinity/libritts_r |
| | --- |
| | |
| | # NeuCodec 🎧 |
| |
|
| | [](https://www.youtube.com/watch?v=O7XH1lGZyYY) |
| |
|
| | *Click the image above to see NeuCodec in action on Youtube!* |
| |
|
| | *Created by Neuphonic - building faster, smaller, on-device voice AI* |
| |
|
| | A lightweight neural codec that encodes audio at just 0.8 kbps - perfect for researchers and builders who need something that *just works* for training high quality text-to-speech models. |
| |
|
| | # Key Features |
| |
|
| | * 🔊 Low bit-rate compression - a speech codec that compresses and reconstructs audio with near-inaudible reconstruction loss |
| | <br> |
| | * 🎼 Upsamples from 16kHz → 24kHz |
| | <br> |
| | * 🌍 Ready for real-world use - train your own SpeechLMs without needing to build your own codec |
| | <br> |
| | * 🏢 Commercial use permitted - use it in your own tools or products |
| | <br> |
| | * 📊 Released with large pre-encoded datasets - we’ve compressed Emilia-YODAS from 1.7TB to 41GB using NeuCodec, significantly reducing the compute requirements needed for training |
| | <br> |
| |
|
| | # Model Details |
| |
|
| | NeuCodec is a Finite Scalar Quantisation (FSQ) based 0.8kbps audio codec for speech tokenization. |
| | It takes advantage of the following features: |
| |
|
| | * FSQ quantisation resulting in a single codebook, making it ideal for downstream modeling with Speech Language Models. |
| | * Trained with CC data such that there are no Non-Commercial data restrictions. |
| | * At 50 tokens/sec and 16 bits per token, the overall bit-rate is 0.8kbps. |
| | * The codec takes in 16kHz input and outputs 24kHz using an upsampling decoder. |
| | * The FSQ encoding scheme allows for bit-level error resistance suitable for unreliable and noisy channels. |
| |
|
| | NeuCodec is largely based on extending the work of [X-Codec2.0](https://huggingface.co/HKUSTAudio/xcodec2). |
| |
|
| | - **Developed by:** Neuphonic |
| | - **Model type:** Neural Audio Codec |
| | - **License:** apache-2.0 |
| | - **Repository:** https://github.com/neuphonic/neucodec |
| | - **Paper:** [arXiv](https://arxiv.org/abs/2509.09550) |
| | - **Pre-encoded Datasets:** |
| | - [Emilia-YODAS-EN](https://huggingface.co/datasets/neuphonic/emilia-yodas-english-neucodec) |
| | - *More coming soon!* |
| |
|
| | # Get Started |
| |
|
| | Use the code below to get started with the model. |
| |
|
| | To install from pypi in a dedicated environment, using Python 3.10 or above: |
| |
|
| | ```bash |
| | conda create -n neucodec python=3.10 |
| | conda activate neucodec |
| | pip install neucodec |
| | ``` |
| | Then, to use in python: |
| |
|
| | ```python |
| | import librosa |
| | import torch |
| | import torchaudio |
| | from torchaudio import transforms as T |
| | from neucodec import NeuCodec |
| | |
| | model = NeuCodec.from_pretrained("neuphonic/neucodec") |
| | model.eval().cuda() |
| | |
| | y, sr = torchaudio.load(librosa.ex("libri1")) |
| | if sr != 16_000: |
| | y = T.Resample(sr, 16_000)(y)[None, ...] # (B, 1, T_16) |
| | |
| | with torch.no_grad(): |
| | fsq_codes = model.encode_code(y) |
| | # fsq_codes = model.encode_code(librosa.ex("libri1")) # or directly pass your filepath! |
| | print(f"Codes shape: {fsq_codes.shape}") |
| | recon = model.decode_code(fsq_codes).cpu() # (B, 1, T_24) |
| | |
| | torchaudio.save("reconstructed.wav", recon[0, :, :], 24_000) |
| | ``` |
| |
|
| | # Training Details |
| |
|
| | The model was trained using the following data: |
| | * Emilia-YODAS |
| | * MLS |
| | * LibriTTS |
| | * Fleurs |
| | * CommonVoice |
| | * HUI |
| | * Additional proprietary set |
| |
|
| | All publically available data was covered by either the CC-BY-4.0 or CC0 license. |
