|
|
--- |
|
|
license: cc-by-4.0 |
|
|
tags: |
|
|
- audio |
|
|
- text-sync |
|
|
- mimi |
|
|
- codec |
|
|
--- |
|
|
|
|
|
# TextSyncMimi-v1 |
|
|
|
|
|
**TextSyncMimi** provides a *text‑synchronous* speech representation designed to plug into LLM‑based speech generation. Instead of operating at a fixed frame rate (time‑synchronous), it represents speech **per text token** and reconstructs high‑fidelity audio through a [Mimi](https://huggingface.co/kyutai/mimi)‑compatible neural audio decoder. |
|
|
|
|
|
> TL;DR: We turn **time‑synchronous** Mimi latents into **text‑synchronous** token latents \([tᵢ, sᵢ]\), then expand them back to Mimi latents and decode to waveform. This makes token‑level control and alignment with LLM text outputs straightforward. |
|
|
|
|
|
## Model overview |
|
|
|
|
|
<div align="center"> |
|
|
<img src="https://i.postimg.cc/V6D84Sxs/Screenshot-2568-08-12-at-16-07-13.png" alt="TextSyncMimi" width="75%" style="margin-left:'auto' margin-right:'auto' display:'block'"/> |
|
|
</div> |
|
|
|
|
|
|
|
|
- **TextSyncMimi components:** |
|
|
- **Cross‑attention encoder** — aligns Mimi’s time‑synchronous sequence (length *T*) to the text sequence (length *N*), producing one continuous speech latent per text token. |
|
|
- **Causal decoder** — expands token‑level latents back to a Mimi‑rate latent sequence suitable for a Mimi decoder. The decoder is streaming. |
|
|
- [Mimi backbone](https://huggingface.co/kyutai/mimi) |
|
|
|
|
|
## Training / Evaluation |
|
|
- **Lossess**: (i) **L2** distance between predicted and ground‑truth continuous Mimi latents, and (ii) **BCE** for the stop token during expansion. |
|
|
- **Training Data**: LibriSpeech (960 hours) + LibriTTS (585 hours) -- around 1.5K hours in total |
|
|
- **Results**: ASR WER on audio reconstructed from different methods (NB: non-zero WER of ground-truth audio came from ASR errors): |
|
|
| Method | Train data | WER ↓ | |
|
|
|------------------|------------------------------------------|------:| |
|
|
| Ground‑truth | – | 2.12 | |
|
|
| Mimi | – | 2.29 | |
|
|
| TASTE | Emilia + LibriTTS | 4.40 | |
|
|
| **TextSyncMimi v1** | **LibriTTS‑R + LibriSpeech** | **3.06** | |
|
|
## Usage |
|
|
|
|
|
### Loading the Model |
|
|
|
|
|
```python |
|
|
from transformers import AutoModel |
|
|
model = AutoModel.from_pretrained("potsawee/TextSyncMimi-v1", trust_remote_code=True) |
|
|
``` |
|
|
|
|
|
See the code of [Speech Editing with TextSync Mimi](https://huggingface.co/spaces/potsawee/TextSyncMimi-SpeechEditing/blob/main/app.py) for a use-case (e.g., encoding, decoding, swapping) of the model |
|
|
|
|
|
|
|
|
## Acknowledgements |
|
|
|
|
|
- Built on top of [Kyutai's Mimi](https://huggingface.co/kyutai/mimi) audio codec |
