README.md

---
license: mit
base_model: microsoft/VibeVoice-ASR-BitNet
pipeline_tag: automatic-speech-recognition
library_name: transformers.js
tags:
- ASR
- bitnet
- onnx
- webgpu
- wasm
- multilingual
languages:
- en
- zh
- fr
- it
- ko
- pt
- vi
---

# VibeVoice-ASR-BitNet · ONNX (WebGPU + WASM)

ONNX export of [microsoft/VibeVoice-ASR-BitNet](https://huggingface.co/microsoft/VibeVoice-ASR-BitNet)
for [🤗 Transformers.js](https://github.com/huggingface/transformers.js) / onnxruntime-web —
multilingual speech recognition with a ternary (1.58-bit) BitNet Qwen2.5-1.5B decoder,
running fully in the browser.

**Live demo:** [multimodalart/vibevoice-asr-bitnet-web](https://huggingface.co/spaces/multimodalart/vibevoice-asr-bitnet-web)

## Layout

The repo follows the transformers.js *audio-text-to-text* (ultravox) layout, so the stock
`UltravoxModel` class drives it with no custom modeling code:

| session | input → output |
|---|---|
| `onnx/audio_encoder*` | `audio_values` [1, samples] (24 kHz mono, −25 dBFS RMS-normalized, length padded to a multiple of 3200) → `audio_features` [1, frames, 1536] |
| `onnx/embed_tokens*` | `input_ids` → `inputs_embeds` |
| `onnx/decoder_model_merged*` | `inputs_embeds` + KV cache → `logits` |

The prompt places one `<|speech_pad|>` (id 151648) per audio frame (`frames = ceil(samples/3200)`);
transformers.js merges `audio_features` into those positions automatically (`audio_token_id` in `config.json`).

## Quantization (mirrors the GGUF scheme)

| component | GGUF (VibeASR.cpp) | this repo | notes |
|---|---|---|---|
| LM projections (196 mats) | I2_S ternary, per-tensor `s = 1/mean\|W\|` | **identical ternary values** in `MatMulNBits` 4-bit blocks (`_q4`, `_q4f16`) and true 2-bit blocks (`_bnb4` file) | bit-exact math, kernels everywhere |
| Embeddings / lm_head | Q6_K | 8-bit per-row / per-column | ≥ Q6_K fidelity |
| VAE encoders | I8_S (int8 weights **and** activations, GELU→ReLU substitution) | int8 per-channel weights, float activations, exact GELU | strictly more accurate |

The released safetensors are BitNet QAT master weights — they only produce sensible output
**after** ternarization, which is applied exactly before export.

## Validation (onnxruntime CPU, greedy, vs ternarized PyTorch reference)

fp32 ONNX: 4/4 transcripts byte-identical. q4 / q4f16 / q2: 3/4 byte-identical; the FLEURS-fr
clip differs by two words ("géographe"→"géologue", "des"→"les"). For comparison, the official
ggml engine (VibeASR.cpp, I8_S+I2_S) transcribes that same clip as *"aux gens ouverts fiscales,
mais assètent sur les toits de douane"* — these ONNX builds are strictly closer to the fp
reference than the original GGUF engine on every tested clip. Full transcripts in
`conversion_report.json`.

## Files / sizes

- WebGPU bundle (`q4f16`): encoder 696 MB + embeddings 234 MB + decoder 869 MB ≈ **1.8 GB**
- WASM bundle (`q4`): ≈ **1.98 GB**
- 2-bit decoder (`decoder_model_merged_bnb4.onnx`, 726 MB): true I2_S-equivalent; current
  onnxruntime CPU 2-bit kernels are much slower than 4-bit — published for experimentation.

## Usage (transformers.js v4)

```js
import { AutoTokenizer, UltravoxModel, Tensor, TextStreamer } from "@huggingface/transformers";

const model_id = "multimodalart/VibeVoice-ASR-BitNet-ONNX";
const tokenizer = await AutoTokenizer.from_pretrained(model_id);
const model = await UltravoxModel.from_pretrained(model_id, {
  device: "webgpu",            // or "wasm"
  dtype: { audio_encoder: "q4f16", embed_tokens: "q4f16", decoder_model_merged: "q4f16" }, // or all "q4"
});

// audio: Float32Array, 24 kHz mono, RMS-normalized to -25 dBFS, zero-padded to length % 3200 == 0
const frames = audio.length / 3200;
const prompt =
  `<|im_start|>system\nYou are a helpful assistant that transcribes audio input into text output in JSON format.<|im_end|>\n` +
  `<|im_start|>user\n<|speech_start|>${"<|speech_pad|>".repeat(frames)}<|speech_end|>\n` +
  `This is a ${duration} seconds audio, please transcribe it.<|im_end|>\n`;
// build ids via tokenizer (or splice numeric ids 151644/151645/151646/151647/151648 directly)

const out = await model.generate({
  ...tokenizer(prompt, { add_special_tokens: false }),
  audio_values: new Tensor("float32", audio, [1, audio.length]),
  max_new_tokens: 512,
  streamer: new TextStreamer(tokenizer, { skip_prompt: true, skip_special_tokens: true }),
});
// The model emits "<|im_start|>assistant\n" first — strip it from the decoded text.
```

## Provenance

Converted on HF Jobs with an open pipeline: official [microsoft/VibeVoice](https://github.com/microsoft/VibeVoice)
modeling code as reference, exact `convert_lm_to_gguf.py` ternarization semantics, custom
static-causal-padding ONNX export of the tokenizer encoders, structural exact-ternary
MatMulNBits packing, transcript-level validation against golden references
([VibeASR.cpp](https://github.com/microsoft/VibeASR.cpp) prompt format).