multimodalart/VibeVoice-ASR-BitNet-ONNX

VibeVoice-ASR-BitNet · ONNX (WebGPU + WASM)

ONNX export of microsoft/VibeVoice-ASR-BitNet for 🤗 Transformers.js / onnxruntime-web — multilingual speech recognition with a ternary (1.58-bit) BitNet Qwen2.5-1.5B decoder, running fully in the browser.

Live demo: multimodalart/vibevoice-asr-bitnet-web

Layout

The repo follows the transformers.js audio-text-to-text (ultravox) layout, so the stock UltravoxModel class drives it with no custom modeling code:

session	input → output
onnx/audio_encoder*	audio_values [1, samples] (24 kHz mono, −25 dBFS RMS-normalized, length padded to a multiple of 3200) → audio_features [1, frames, 1536]
onnx/embed_tokens*	input_ids → inputs_embeds
onnx/decoder_model_merged*	inputs_embeds + KV cache → logits

The prompt places one <|speech_pad|> (id 151648) per audio frame (frames = ceil(samples/3200)); transformers.js merges audio_features into those positions automatically (audio_token_id in config.json).

Quantization (mirrors the GGUF scheme)

component	GGUF (VibeASR.cpp)	this repo	notes
LM projections (196 mats)	I2_S ternary, per-tensor s = 1/mean|W|	identical ternary values in MatMulNBits 4-bit blocks (_q4, _q4f16) and true 2-bit blocks (_bnb4 file)	bit-exact math, kernels everywhere
Embeddings / lm_head	Q6_K	8-bit per-row / per-column	≥ Q6_K fidelity
VAE encoders	I8_S (int8 weights and activations, GELU→ReLU substitution)	int8 per-channel weights, float activations, exact GELU	strictly more accurate

The released safetensors are BitNet QAT master weights — they only produce sensible output after ternarization, which is applied exactly before export.

Validation (onnxruntime CPU, greedy, vs ternarized PyTorch reference)

fp32 ONNX: 4/4 transcripts byte-identical. q4 / q4f16 / q2: 3/4 byte-identical; the FLEURS-fr clip differs by two words ("géographe"→"géologue", "des"→"les"). For comparison, the official ggml engine (VibeASR.cpp, I8_S+I2_S) transcribes that same clip as "aux gens ouverts fiscales, mais assètent sur les toits de douane" — these ONNX builds are strictly closer to the fp reference than the original GGUF engine on every tested clip. Full transcripts in conversion_report.json.

Files / sizes

• WebGPU bundle (q4f16): encoder 696 MB + embeddings 234 MB + decoder 869 MB ≈ 1.8 GB
• WASM bundle (q4): ≈ 1.98 GB
• 2-bit decoder (decoder_model_merged_bnb4.onnx, 726 MB): true I2_S-equivalent; current onnxruntime CPU 2-bit kernels are much slower than 4-bit — published for experimentation.

Usage (transformers.js v4)

import { AutoTokenizer, UltravoxModel, Tensor, TextStreamer } from "@huggingface/transformers";

const model_id = "multimodalart/VibeVoice-ASR-BitNet-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const model = await UltravoxModel.from_pretrained(model_id, {
  device: "webgpu",            // or "wasm"
  dtype: { audio_encoder: "q4f16", embed_tokens: "q4f16", decoder_model_merged: "q4f16" }, // or all "q4"
});

// audio: Float32Array, 24 kHz mono, RMS-normalized to -25 dBFS, zero-padded to length % 3200 == 0
const frames = audio.length / 3200;
const prompt =
  `<|im_start|>system\nYou are a helpful assistant that transcribes audio input into text output in JSON format.<|im_end|>\n` +
  `<|im_start|>user\n<|speech_start|>${"<|speech_pad|>".repeat(frames)}<|speech_end|>\n` +
  `This is a ${duration} seconds audio, please transcribe it.<|im_end|>\n`;
// build ids via tokenizer (or splice numeric ids 151644/151645/151646/151647/151648 directly)

const out = await model.generate({
  ...tokenizer(prompt, { add_special_tokens: false }),
  audio_values: new Tensor("float32", audio, [1, audio.length]),
  max_new_tokens: 512,
  streamer: new TextStreamer(tokenizer, { skip_prompt: true, skip_special_tokens: true }),
});
// The model emits "<|im_start|>assistant\n" first — strip it from the decoded text.

Provenance

Converted on HF Jobs with an open pipeline: official microsoft/VibeVoice modeling code as reference, exact convert_lm_to_gguf.py ternarization semantics, custom static-causal-padding ONNX export of the tokenizer encoders, structural exact-ternary MatMulNBits packing, transcript-level validation against golden references (VibeASR.cpp prompt format).