VoiceChat 11B — MLX 8-bit

NVIDIA's complete duplex speech-to-speech NemotronLabs VoiceChat 11B pipeline, converted to MLX for Apple Silicon: continuous user-speech perception, the Nemotron-H text/function channels, direct EAR-TTS agent-speech generation, and the audio-codec decoder.

The upstream NVIDIA model card cites SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Model (Interspeech 2025) as its primary architecture reference.

The repository name retains Perception for compatibility with the original understanding-only release. The current payload is the complete encoder/ + llm/ + tts/ bundle; strict full-pipeline loaders reject an older download that does not contain tts/.

Scope. Built-in Aria speech generation is verified end to end. The codec encoder used for voice cloning is included for checkpoint completeness but is not verified and does not currently reconstruct real audio correctly. Use the shipped Aria prompt; do not advertise custom-voice cloning.

Performance status. The runtime is functionally streaming, but the full speech-to-speech path does not yet sustain the 80 ms frame budget on the tested Apple Silicon system. Model-timeline turn positions are not wall-clock latency. Natural turn-taking latency has not yet been established against an upstream reference, so forced-BOS regression timings must not be cited as natural latency.

The current Swift release benchmark on an M5 Pro (48 GB) feeds one 80 ms live input frame per push over a controlled 120-frame fixture:

Variant	Peak RSS	First spoken text token	First playable audio	Total/frame p50 / p95	Whole-pipeline RTF
INT8	12.21 GB	68.8 ms	105.1 ms	104.6 / 114.0 ms	1.34
INT5	8.73 GB	57.5 ms	91.4 ms	93.0 / 104.7 ms	1.17

Both variants remain slower than real time once perception is included. The first-token/audio figures are hot compute latency for the first spoken response frame, not learned turn-taking latency. Peak macOS physical footprint was 24.42 GB for INT8 and 20.94 GB for INT5; it includes file-backed MLX mappings that RSS can undercount.

Model


Parameters included	11.1 B (complete checkpoint)
Language backbone	Nemotron-H hybrid — 56 layers, 27 Mamba2 / 25 MLP / 4 attention, hidden 4480
Speech encoder	Streaming FastConformer — 24 layers, d_model 1024, 8 heads
Speech decoder	EAR-TTS — 28-layer Gemma3 backbone, MaskGIT-style 8-iteration generation
Audio codec	31-stage RVQ, 1024 codes/stage, 22.05 kHz, 80 ms frames
Quantization	8-bit, affine, group size 64
Format	MLX safetensors
Total size	12.13 GB
Audio input	16 kHz mono, 80 ms frames
Context length	131 072 tokens
Attention context	70 frames left, 0 right (streaming)

Files

File	Size	Description
`llm/model.safetensors`	10.09 GB	Nemotron-H backbone, embeddings, LM head, tool-call head
`llm/config.json`	—	Architecture, hybrid layer pattern, quantization spec
`llm/tokenizer.json`	17 MB	131 072-entry tokenizer
`encoder/model.safetensors`	744 MB	FastConformer, modality projection, RNNT decoder + joint
`encoder/config.json`	—	Encoder geometry and streaming attention context
`tts/model.safetensors`	1.30 GB	EAR-TTS, Aria prompt, RVQ embeddings, and dense fp16 codec
`tts/config.json`	—	Speech geometry, generation defaults, per-tensor quantization
`voicechat_mlx.py`	—	Understanding-path loader
`tts_mlx.py`, `codec_mlx.py`	—	Speech decoder and codec loaders
`speak.py`, `render_dialog.py`	—	Text-to-speech and full-duplex runnable examples

Performance

Measured against the fp16 bundle these were quantized from, teacher-forced over an identical corpus. Agreement is how often the quantized model picks the same next token as fp16; KL is the divergence of the full next-token distribution (lower is better for both).

Variant	Size	Top-1 agreement vs fp16	KL (nats)
fp16 reference	22.19 GB	—	—
8-bit	12.11 GB	100.00%	0.00018
5-bit (LLM and MoG heads at 8-bit)	8.56 GB	92.55%	0.01213

This variant: 100.00% agreement, KL 0.00018.

8-bit reproduces the fp16 model's greedy output token for token. 5-bit diverges on roughly one token in thirteen — acceptable for conversational text, but worth measuring on your own task before relying on it for structured output such as tool-call arguments, where a single divergent token invalidates the result.

Perplexity is deliberately not quoted: on a short corpus, quantization noise can lower it without the model being better, so agreement and KL are the honest measures here.

Usage

Swift — complete duplex speech-to-speech session

import VoiceChat

let model = try await VoiceChatModel.loadFromHub(
    "aufklarer/VoiceChat-11B-Perception-MLX-int8")
let session = try await model.startSession()

for event in try await session.pushAudio(mono16kSamples) {
    playback.enqueue(
        event.audio,
        sampleRate: VoiceChatSession.outputSampleRate)
}

// Let a response that starts near the end of a finite clip finish.
_ = try await session.pushSilence(seconds: 6)
print(await session.userTranscript())
print(await session.reply())

Each event contains the text/function decisions and exactly one 1,764-sample, 22.05 kHz output frame. EAR-TTS uses the checkpoint's text control roles and eight MaskGIT unmasking iterations; it is not a projection from the 4,480-wide language-model hidden state.

Python / MLX

import mlx.core as mx
from voicechat_mlx import load_llm, load_perception

# Speech understanding: log-mel -> language-model embedding space
encode = load_perception("encoder")
embeddings, lengths = encode(log_mel)     # (B, T, 128) -> (B, T/8, 4480)

# Language backbone
model, tokenizer = load_llm("llm")
ids = mx.array(tokenizer.encode("The capital of Norway is"))
logits = model(ids[None])
print(tokenizer.decode([int(mx.argmax(logits[0, -1]))]))

pip install mlx mlx-lm parakeet-mlx huggingface_hub
hf download aufklarer/VoiceChat-11B-Perception-MLX-int8 --local-dir ./voicechat
python -c "from voicechat_mlx import load_llm; m, t = load_llm('./voicechat/llm'); print('ok')"

Generate the built-in Aria voice, or render both sides of a duplex exchange:

python speak.py --checkpoint . --tokenizer llm --text "Hello from VoiceChat" --out hello.wav
python render_dialog.py --bundle . --audio user.wav --out dialog.wav

Every published speech payload must first pass the canonical silence check:

python codec_mlx.py --checkpoint . --verify-silence

The encoder is a NeMo streaming FastConformer and differs from a stock Conformer in three ways that all fail silently if ignored — no biases on the feed-forward, attention and convolution linears; a LayerNorm in place of the convolution BatchNorm; and causal subsampling that keeps 17 frequency bins rather than 16. voicechat_mlx.py handles all three, so use it rather than constructing the module tree yourself.

Source

Converted from nvidia/NVIDIA-NemotronLabs-VoiceChat-11B. The language backbone derives from nvidia/NVIDIA-Nemotron-Nano-9B-v2, which also supplies the tokenizer. Licensed under OpenMDW 1.1.