VoiceChat 11B Perception — MLX 5-bit (output heads 8-bit)

The speech encoder and language backbone of NVIDIA's NemotronLabs VoiceChat 11B, converted to MLX for Apple Silicon.

What this is, and what it is not. This bundle contains the parts that understand speech: the streaming FastConformer encoder, the bridge into the language model, the RNNT transcript head, and the full 9.5 B-parameter Nemotron-H backbone. It does not contain the speech synthesis half — the TTS decoder, the audio codec, and the full-duplex loop are not included, so this cannot hold a spoken conversation on its own. It is a speech understanding front-end plus a language model, not a voice agent.

Model


Parameters included	10.1 B of 11.1 B (all but TTS decoder + codec)
Language backbone	Nemotron-H hybrid — 56 layers, 27 Mamba2 / 25 MLP / 4 attention, hidden 4480
Speech encoder	Streaming FastConformer — 24 layers, d_model 1024, 8 heads
Quantization	5-bit (output heads 8-bit), affine, group size 64
Format	MLX safetensors
Total size	7.52 GB
Audio input	16 kHz mono, 80 ms frames
Context length	131 072 tokens
Attention context	70 frames left, 0 right (streaming)

Files

File	Size	Description
`llm/model.safetensors`	6.97 GB	Nemotron-H backbone, embeddings, LM head, tool-call head
`llm/config.json`	—	Architecture, hybrid layer pattern, quantization spec
`llm/tokenizer.json`	17 MB	131 072-entry tokenizer
`encoder/model.safetensors`	543 MB	FastConformer, modality projection, RNNT decoder + joint
`encoder/config.json`	—	Encoder geometry and streaming attention context
`voicechat_mlx.py`	—	Self-contained loader

Performance

Measured against the fp16 bundle these were quantized from, teacher-forced over an identical corpus. Agreement is how often the quantized model picks the same next token as fp16; KL is the divergence of the full next-token distribution (lower is better for both).

Variant	Size	Top-1 agreement vs fp16	KL (nats)
fp16 reference	20.2 GB	—	—
8-bit	10.8 GB	100.00%	0.00018
5-bit (heads at 8-bit)	7.5 GB	92.55%	0.01213

This variant: 92.55% agreement, KL 0.01213.

8-bit reproduces the fp16 model's greedy output token for token. 5-bit diverges on roughly one token in thirteen — acceptable for conversational text, but worth measuring on your own task before relying on it for structured output such as tool-call arguments, where a single divergent token invalidates the result.

Perplexity is deliberately not quoted: on a short corpus, quantization noise can lower it without the model being better, so agreement and KL are the honest measures here.

Usage

import mlx.core as mx
from voicechat_mlx import load_llm, load_perception

# Speech understanding: log-mel -> language-model embedding space
encode = load_perception("encoder")
embeddings, lengths = encode(log_mel)     # (B, T, 128) -> (B, T/8, 4480)

# Language backbone
model, tokenizer = load_llm("llm")
ids = mx.array(tokenizer.encode("The capital of Norway is"))
logits = model(ids[None])
print(tokenizer.decode([int(mx.argmax(logits[0, -1]))]))

pip install mlx mlx-lm parakeet-mlx huggingface_hub
hf download aufklarer/VoiceChat-11B-Perception-MLX-int5 --local-dir ./voicechat
python -c "from voicechat_mlx import load_llm; m, t = load_llm('./voicechat/llm'); print('ok')"

The encoder is a NeMo streaming FastConformer and differs from a stock Conformer in three ways that all fail silently if ignored — no biases on the feed-forward, attention and convolution linears; a LayerNorm in place of the convolution BatchNorm; and causal subsampling that keeps 17 frequency bins rather than 16. voicechat_mlx.py handles all three, so use it rather than constructing the module tree yourself.

Source

Converted from nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The language backbone derives from nvidia/NVIDIA-Nemotron-Nano-9B-v2, which also supplies the tokenizer. Licensed under OpenMDW 1.1.