---
license: mit
tags:
  - audio
  - speaker-embedding
  - voice-cloning
  - moshi
  - tts
language:
  - en
  - fr
library_name: transformers
base_model: kyutai/tts-1.6b-en_fr
---

# Unmute Encoder

A speaker embedding encoder trained to replicate Kyutai's unreleased "unmute encoder".
This model extracts speaker embeddings from audio for use with Kyutai's Moshi TTS system.

## Model Description

The encoder is built on top of Kyutai's Mimi neural audio codec:

1. **Mimi Encoder**: Frozen Mimi encoder extracts latent audio representations
2. **MLP Projector**: Trainable MLP head projects Mimi's latents to the target embedding space
3. **Output**: Speaker embeddings of shape `[512, 125]` (512 channels, 125 time steps for 10s audio)

```
Audio (24kHz, 10s) -> Mimi Encoder -> Latent [512, T] -> MLP Projector -> Embedding [512, 125]
```

## Usage

```python
from src.models.mimi import MimiEncoder

# Load the encoder
encoder = MimiEncoder.from_pretrained(
    model_name="jspaulsen/unmute-encoder",
    device="cuda",
    num_codebooks=32,
)

# Create embedding from audio tensor [1, 1, T] at 24kHz
output = encoder(audio_tensor)
embedding = output.embedding  # [1, 512, 125]
```

## Training

Trained using supervised learning with a hybrid loss (L1 + cosine similarity) against
speaker embeddings from [kyutai/tts-voices](https://huggingface.co/kyutai/tts-voices).


### Training Details

- **Global step**: 500
- **Epoch**: 83.33333333333333
- **Best metric**: 0.17919586598873138


## Acknowledgments

- [Kyutai](https://kyutai.org/) for releasing the Moshi TTS models and speaker embeddings