eustlb/pocket-tts-mimi-decoder-only

Pocket TTS codec (Mimi), decoder only

The decoder of the audio codec of Kyutai's Pocket TTS in 🤗 Transformers format (PocketTTSAudioTokenizerModel without its encoder): it turns 12.5 Hz latents into 24 kHz audio. Like kyutai/pocket-tts-without-voice-cloning, it cannot encode audio, so it cannot clone a voice; the full codec is eustlb/pocket-tts-mimi (gated).

Weights: the decoder-side mimi.* tensors of kyutai/pocket-tts, converted with convert_pocket_tts_to_hf.py. Original code: kyutai-labs/pocket-tts.

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("eustlb/pocket-tts-mimi-decoder-only", device_map="auto")
latents = torch.randn(1, 25, 32, device=model.device)  # (batch, frames, latent_dim), e.g. Pocket TTS's generated latents
audio_values = model.decode(latents).audio_values

See the Transformers docs.

Prohibited use

Use of our model must comply with all applicable laws and regulations and must not result in, involve, or facilitate any illegal, harmful, deceptive, fraudulent, or unauthorized activity. Prohibited uses include, without limitation, voice impersonation or cloning without explicit and lawful consent; misinformation, disinformation, or deception (including fake news, fraudulent calls, or presenting generated content as genuine recordings of real people or events); and the generation of unlawful, harmful, libelous, abusive, harassing, discriminatory, hateful, or privacy-invasive content. We disclaim all liability for any non-compliant use.

Authors

Manu Orsini*, Simon Rouard*, Gabriel De Marmiesse*, Václav Volhejn, Neil Zeghidour, Alexandre Défossez

*equal contribution