pocket-tts-mlx-language-converter
Converts the non-English kyutai/pocket-tts language checkpoints into the layout that mlx-audio can load.
No weights are redistributed here. kyutai/pocket-tts is a gated repository —
accept their conditions and download it yourself.
Source and issues: https://github.com/OpenMinded-Labs/pocket-tts-mlx-language-converter
The problem
Kyutai ships language checkpoints (German, French, Italian, Spanish,
Portuguese) under languages/ in their original layout. mlx-audio implements
the converted layout published as mlx-community/pocket-tts, which covers
English only. Pointing mlx-audio at a language checkpoint fails on tensor
names, then on shapes.
Usage
pip install mlx mlx-audio huggingface_hub numpy
huggingface-cli login # the source repo is gated
python convert_language.py german
python load_speaker.py german lola "Ein Satz zum Testen." out.wav
Available: german, french, italian, spanish, portuguese, plus the
deeper *_24l variants (pass --layers 24 to load_speaker.py).
What it does
Tensor names. The two layouts name the Mimi modules differently
(mimi.decoder.model.0.conv.weight vs
mimi.decoder.init_conv1d.conv.conv.weight). The mapping is not hardcoded
from module structure — it is derived. The English weights exist in both
layouts, so the same numbers appear under both names; matching by value
fingerprint gives the mapping and the per-tensor operation (identity,
reshape, or axis permutation). One trap worth naming: a byte fingerprint
ignores shape, so (64,1,7) and (64,7,1) hash identically. The required
operation is therefore determined by comparing each pair separately.
Latent width. Exactly two tensors differ in shape from English:
mimi.downsample.conv.conv.weight lang (32,512,32) en (512,512,32)
flow_lm.speaker_proj_weight lang (1024,32) en (1024,512)
encode_to_latent() does not use the quantizer; the latent comes from
downsample, and the language checkpoints project 512 → 32 in that step,
while mlx-audio's ConvDownsample1d uses a single dim for both sides. The
converter zero-pads the downsample output channels to 512 and pads the
speaker projection to match, so that
latent512 = [latent32 | 0]
W512 = [W32 | 0]
latent512 @ W512.T == latent32 @ W32.T
This is exact, not an approximation, and the script verifies it (maximum deviation must be 0) before writing anything. No model code is patched.
Layer count. The *_24l variants have 24 transformer layers where
English has 6; the derived mapping is generalised over the layer index.
What it does not do
Voice cloning from your own audio does not work. The language checkpoints
are a newer generation than the one mlx-audio implements: their bundled
speaker files are pre-filled transformer KV caches, not the audio_prompt
latents the English release uses. mlx-audio's ref_audio path builds state
through _encode_audio(), which these checkpoints do not use — generation
terminates after a fraction of a second.
Use the bundled speakers instead (languages/<lang>/embeddings/, 26 of them
for German). load_speaker.py loads one directly into the model state; the
only adjustment needed is the axis order, since the file stores
(batch, position, head, dim) and mlx-audio wants
(batch, head, position, dim).
Supporting cloning would mean implementing that newer conditioning path in mlx-audio. That is a port, not a weight remap, and is out of scope here.
Unconditioned generation is unreliable. With an empty state the 6-layer German model stops almost immediately; the 24-layer one produces audio but of poor quality. Pass a speaker.
Verifying output
Generation stopping early is the failure mode to watch for, and it is silent
— you get a valid WAV file containing a fraction of the text. load_speaker.py
prints the duration against a rough expectation (~15 characters of text per
second of speech) and warns when it falls far short. Worth keeping if you
build on this.
Tested with
macOS 15 on Apple M4, mlx-audio 0.2.x, Python 3.13. German 6-layer and
24-layer checkpoints. The other languages use the same layout but have not
been listened to.
Licence
The scripts are MIT. The weights they convert are kyutai's, CC-BY-4.0, and subject to the access conditions of their repository.