---
license: apache-2.0
language:
- en
- hi
- bn
- gu
- kn
- ml
- mr
- or
- pa
- ta
- te
tags:
- audio
- speech
- audio-codec
- speech-tokenizer
- speech-language-models
pipeline_tag: audio-to-audio
datasets:
- amphion/Emilia-Dataset
- mythicinfinity/libritts
- mythicinfinity/libritts_r
- CSTR-Edinburgh/vctk
- ai4bharat/Rasa
- ai4bharat/indicvoices_r
- ai4bharat/IndicVoices
---

# IndiCodec

[Audio samples](https://codeprakhar25.github.io/IndiCodec/) · [Code on GitHub](https://github.com/codeprakhar25/IndiCodec)

A 1.09 kbps neural speech codec for building speech language models and TTS. It turns speech into two streams of
integer tokens and decodes them back to 24 kHz audio.

It was trained mostly on English and 10 Indian languages, and it is strongest on Indian languages: among codecs at
about 1 kbps, it keeps the most words in every one of the 10 we tested.

## Key features

- **Low bitrate:** 1.09 kbps, 75 tokens per second.
- **Two token streams:** a content stream (25 Hz) that carries the words, and an acoustic stream (50 Hz) that carries
  the voice and detail.
- **Split FSQ, not RVQ:** each stream is a single FSQ grid. Content is coded first, and one acoustic grid codes what
  remains, instead of a stack of residual codebooks. Each frame is one id per stream, with no learned codebook.
- **16 kHz in, 24 kHz out:** any input sample rate is resampled to 16 kHz.
- **One file:** all weights are in `model.safetensors`. Nothing else is downloaded.
- **Commercial use allowed:** Apache 2.0. All training data is under open licences, none of them non-commercial.

## Model details

| | |
|---|---|
| Content stream | 25 Hz, FSQ 8 × 3 levels = 6,561 ids, 317 bps |
| Acoustic stream | 50 Hz, FSQ 6 × 6 levels = 46,656 ids, 775 bps |
| Bitrate | 1.09 kbps |
| Parameters | 624 M in the file, 387 M of them w2v-BERT 2.0; decoding alone uses 186 M |
| Licence | Apache 2.0 |

The content stream quantises w2v-BERT 2.0 features (layer 16). The acoustic stream quantises what a CNN encoder hears
beyond the content stream. A transformer decoder with an iSTFT head turns the sum back into audio.

IndiCodec is built on [NeuCodec](https://huggingface.co/neuphonic/neucodec) by Neuphonic: it uses NeuCodec's
architecture, and training started from its released weights. The two-stream quantiser, the encoder training and the
training recipe are new.

## Quality

Speech codecs at about 1 kbps, on speakers not seen in training. Every codec is scored on the same clips.

PESQ-WB (higher is better):

| Test set | IndiCodec<br>1.09 kbps | Mimi, 8 codebooks<br>1.10 kbps | SNAC 24 kHz<br>0.98 kbps | NeuCodec<br>0.80 kbps |
|---|---|---|---|---|
| LibriTTS-R test (English) | **2.41** | 2.32 | 2.29 | 2.19 |
| CMU-Arctic (English) | **2.30** | 2.28 | 2.17 | 2.06 |
| Seed-TTS eval (English) | 1.99 | **2.07** | 1.82 | 1.81 |
| FLEURS, 10 Indian languages | **1.90** | 1.84 | 1.69 | 1.69 |

Words after a round trip (lower is better):

| | IndiCodec | Mimi | SNAC | NeuCodec |
|---|---|---|---|---|
| English WER, Seed-TTS eval | **2.5 %** | 3.6 % | 4.9 % | 3.5 % |
| English WER, CMU-Arctic | **2.3 %** | 3.2 % | 3.8 % | 2.8 % |
| Indian-language CER, FLEURS, 10 languages | **6.3 %** | 14.3 % | 13.3 % | 8.9 % |
| Tokens per second | 75 | 100 | 82 | 50 |

English WER uses Whisper large-v3. Indian-language CER uses the IndicConformer 600M ASR; the original recordings score
5.2 %, so IndiCodec adds 1.0 point. The 10 languages are Hindi, Bengali, Gujarati, Kannada, Malayalam, Marathi, Odia,
Punjabi, Tamil and Telugu.

## Use

```bash
hf download prakharkhatri/IndiCodec --local-dir IndiCodec && cd IndiCodec
pip install torch==2.6.0 torchaudio==2.6.0 safetensors soundfile transformers==4.51.* vector-quantize-pytorch==1.17.8 torchtune==0.5.0 torchao==0.9.0 local-attention einops librosa
pip install --no-deps "git+https://github.com/neuphonic/neucodec@ed3e6cd"
```

```python
import soundfile as sf
from indicodec import IndiCodec

codec = IndiCodec.load("model.safetensors")
wav, sr = sf.read("speech.wav")

content, acoustic = codec.encode(wav, sr)    # token arrays
audio = codec.decode(content, acoustic)      # 24 kHz
```

## Training data

English: Emilia-YODAS (English part only), LibriTTS, LibriTTS-R and VCTK. Indian languages: Rasa, IndicVoices-R and
IndicVoices from AI4Bharat. Plus about 15 h of laughs, sighs and coughs from VocalSound. All CC BY 4.0, except
VocalSound (CC BY-SA 4.0). No audio is included in this release.

## Limits

- Speech only; not for music.
- Not streaming: the encoder and the decoder both attend over the whole clip.

## Licence

Apache 2.0. Built on [NeuCodec](https://huggingface.co/neuphonic/neucodec) (Apache 2.0). The bundled w2v-BERT 2.0
layers are Meta's, under MIT; its licence text is in `indicodec/w2v_bert/LICENSE`.