Duplicate from microsoft/VibeVoice-ASR-BitNet

Co-authored-by: Furu Wei <frontierai@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,5 @@
+*.gguf filter=lfs diff=lfs merge=lfs -text
+model-00001-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00002-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text
+model-00003-of-00003.safetensors filter=lfs diff=lfs merge=lfs -text
+report_overview.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,124 @@
+---
+language:
+- en
+- zh
+- fr
+- it
+- ko
+- pt
+- vi
+license: mit
+pipeline_tag: automatic-speech-recognition
+tags:
+- ASR
+- quantization
+- cpu-inference
+- gguf
+- bitnet
+- multilingual
+library_name: ggml
+---
+
+## VibeVoice-ASR-BitNet
+
+[![GitHub](https://img.shields.io/badge/GitHub-Repo-black?logo=github)](https://github.com/microsoft/VibeASR.cpp)
+[![Technical Report](https://img.shields.io/badge/arXiv-2607.21075-b31b1b?logo=arxiv)](https://arxiv.org/abs/2607.21075)
+[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
+
+**VibeVoice-ASR-BitNet** is a compressed variant of [VibeVoice-ASR](https://huggingface.co/microsoft/VibeVoice-ASR) optimized for **real-time inference on edge CPUs** — no GPU required. Through heterogeneous quantization, the model is compressed from 4.62 GB to **1.58 GB** while achieving **1.6–2.3× faster** inference than Whisper.cpp with real-time capability (RTF < 1) on as few as 3 CPU threads.
+
+➡️ **Code:** [microsoft/VibeASR.cpp](https://github.com/microsoft/VibeASR.cpp)<br>
+➡️ **Report:** [VibeVoice-ASR-BitNet Technical Report](https://arxiv.org/abs/2607.21075)<br>
+➡️ **Base Model:** [microsoft/VibeVoice-ASR](https://huggingface.co/microsoft/VibeVoice-ASR)<br>
+
+<p align="center">
+  <img src="figures/report_overview.png" width="90%"/>
+</p>
+
+---
+
+## 🔥 Key Features
+
+- **⚡ Real-time on CPU** — RTF < 1 with 3+ threads on commodity x86 (AVX2) and ARM (NEON) hardware
+- **📦 Compact** — 1.58 GB total (2.9× compression from FP16), fits in edge device memory
+- **🌍 Multilingual** — English, Chinese, French, Italian, Korean, Portuguese, Vietnamese, and more
+- **🔧 Custom SIMD Kernels** — Fused operators within the ggml framework for both ARM and x86 platforms
+
+---
+
+## Quantization Strategy
+
+<div align="center">
+
+| Component | FP16 | Quantized | Method | Compression |
+|:-:|:-:|:-:|:-:|:-:|
+| VAE Tokenizer | 1.31 GB | 0.65 GB | I8\_S | 2.0× |
+| LM Decoder | 3.32 GB | 0.92 GB | I2\_S + Q6\_K | 3.6× |
+| **Total** | **4.62 GB** | **1.58 GB** | — | **2.9×** |
+
+</div>
+
+---
+
+## Evaluation
+
+### Inference Speed
+
+<div align="center">
+
+| Threads | 1 | 2 | 3 | 4 | 6 | 8 |
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| RTF | 1.98 | 1.08 | **0.77** | **0.63** | **0.49** | **0.42** |
+| vs. Whisper.cpp | 2.28× | 2.12× | 1.86× | 1.86× | 1.71× | 1.55× |
+
+</div>
+
+> Benchmarked on AMD EPYC 7V13 (AVX2+FMA) with 20s audio. **Bold** = RTF < 1 (real-time).
+
+### Accuracy (WER%)
+
+<div align="center">
+
+| Benchmark | VibeVoice-ASR-7B | VibeVoice-ASR-BitNet | Parakeet | Whisper | SenseVoice | FunASR |
+|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
+| MLC-EN | 7.82 | **8.25** | 8.40 | 13.57 | 12.39 | 11.36 |
+| MLC-FR | 16.03 | 17.41 | — | — | — | — |
+| MLC-IT | 15.67 | 17.23 | — | — | — | — |
+| MLC-KO | 9.83 | 11.15 | — | — | — | — |
+| MLC-PT | 22.41 | 24.87 | — | — | — | — |
+| MLC-VI | 20.15 | 22.38 | — | — | — | — |
+| AISHELL4 | 19.83 | 27.45 | — | — | 22.52 | **20.41** |
+| AMI-ihm | 17.42 | **21.36** | 21.92 | 27.07 | 30.81 | 32.07 |
+| AMI-sdm | 24.18 | **25.87** | 26.33 | 36.92 | 48.11 | 40.17 |
+| AliMeeting | 36.21 | 40.58 | — | — | **38.75** | 39.27 |
+| Fleurs-en | 4.73 | 5.21 | 4.09 | **3.99** | 6.84 | 4.93 |
+| Fleurs-zh | 7.92 | 8.35 | — | — | **5.56** | 7.00 |
+| Libri-clean | 2.17 | 2.41 | **1.49** | 1.98 | 2.78 | 1.58 |
+| Libri-other | 5.84 | 6.27 | **3.13** | 3.60 | 6.81 | 4.01 |
+| VoxPopuli | 4.92 | **5.18** | 5.26 | 7.19 | 8.63 | 6.46 |
+
+</div>
+
+---
+
+## Model Files
+
+<div align="center">
+
+| File | Size | Description |
+|:-:|:-:|:-:|
+| `vibeasr-vae-encoder-i8_s.gguf` | 0.65 GB | VAE tokenizer, I8\_S quantized (ready to use) |
+| `vibeasr-lm-i2_s-embed-q6_k.gguf` | 0.92 GB | LM decoder, I2\_S quantized (ready to use) |
+| `model-*.safetensors` | 10.7 GB | Original SafeTensors (for conversion) |
+
+</div>
+
+---
+
+## License
+
+This project is licensed under the MIT License.
+
+## Contact
+
+This project was conducted by members of Microsoft Research. If you have suggestions, questions, or observe unexpected behavior, please contact us at VibeVoice@microsoft.com.

config.json

@@ -0,0 +1,148 @@
+{
+  "_attn_implementation_autoset": false,
+  "acoustic_tokenizer_config": {
+    "causal": true,
+    "channels": 1,
+    "conv_bias": true,
+    "conv_norm": "none",
+    "corpus_normalize": 0.0,
+    "decoder_depths": null,
+    "decoder_n_filters": 32,
+    "decoder_ratios": [
+      8,
+      5,
+      5,
+      4,
+      2,
+      2
+    ],
+    "disable_last_norm": true,
+    "dtype": "bfloat16",
+    "encoder_depths": "3-3-3-3-3-3-8",
+    "encoder_n_filters": 32,
+    "encoder_ratios": [
+      8,
+      5,
+      5,
+      4,
+      2,
+      2
+    ],
+    "fix_std": 0.5,
+    "layer_scale_init_value": 1e-06,
+    "layernorm": "RMSNorm",
+    "layernorm_elementwise_affine": true,
+    "layernorm_eps": 1e-05,
+    "mixer_layer": "depthwise_conv",
+    "model_type": "vibevoice_acoustic_tokenizer",
+    "pad_mode": "constant",
+    "std_dist_type": "gaussian",
+    "vae_dim": 64,
+    "weight_init_value": 0.01
+  },
+  "acoustic_vae_dim": 64,
+  "architectures": [
+    "VibeVoiceForASRTraining"
+  ],
+  "decoder_config": {
+    "attention_dropout": 0.0,
+    "dtype": "bfloat16",
+    "hidden_act": "silu",
+    "hidden_size": 1536,
+    "initializer_range": 0.02,
+    "intermediate_size": 8960,
+    "layer_types": [
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention",
+      "full_attention"
+    ],
+    "max_position_embeddings": 65536,
+    "max_window_layers": 28,
+    "model_type": "qwen2",
+    "num_attention_heads": 12,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 2,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000.0,
+    "sliding_window": null,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "use_sliding_window": false,
+    "vocab_size": 151936
+  },
+  "diffusion_head_config": {
+    "ddpm_batch_mul": 4,
+    "ddpm_beta_schedule": "cosine",
+    "ddpm_num_inference_steps": 20,
+    "ddpm_num_steps": 1000,
+    "diffusion_type": "ddpm",
+    "head_ffn_ratio": 3.0,
+    "head_layers": 4,
+    "hidden_size": 1536,
+    "latent_size": 64,
+    "model_type": "vibepod_diffusion_head",
+    "prediction_type": "v_prediction",
+    "rms_norm_eps": 1e-05,
+    "speech_vae_dim": 64
+  },
+  "dtype": "float32",
+  "model_type": "vibevoice",
+  "semantic_tokenizer_config": {
+    "causal": true,
+    "channels": 1,
+    "conv_bias": true,
+    "conv_norm": "none",
+    "corpus_normalize": 0.0,
+    "disable_last_norm": true,
+    "dtype": "bfloat16",
+    "encoder_depths": "3-3-3-3-3-3-8",
+    "encoder_n_filters": 32,
+    "encoder_ratios": [
+      8,
+      5,
+      5,
+      4,
+      2,
+      2
+    ],
+    "fix_std": 0,
+    "layer_scale_init_value": 1e-06,
+    "layernorm": "RMSNorm",
+    "layernorm_elementwise_affine": true,
+    "layernorm_eps": 1e-05,
+    "mixer_layer": "depthwise_conv",
+    "model_type": "vibevoice_semantic_tokenizer",
+    "pad_mode": "constant",
+    "std_dist_type": "none",
+    "vae_dim": 128,
+    "weight_init_value": 0.01
+  },
+  "semantic_vae_dim": 128,
+  "transformers_version": "4.57.6"
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": false,
+  "eos_token_id": 151643,
+  "max_new_tokens": 2048,
+  "transformers_version": "4.37.0"
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:58cb328634bb4b7e5afcc4f14c43261a0c636c9031b0b085bd3ef53c131aaf19
+size 4996674400

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:410349401256a9a995424408f8f77a21cad5e53183e07207820d7b25e84682df
+size 4963184724

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49c2d7591f9dcbbd6fb37baa58379211f21df30dbd3c6ff39cb8c81473660cdb
+size 1296761656

tokenizer_config.json

@@ -0,0 +1,40 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": ["<|im_start|>", "<|im_end|>"],
+  "bos_token": null,
+  "chat_template": "{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n' }}{% endif %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

vibeasr-lm-i2_s-embed-q6_k.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fbe273d8dc2f2433bb25f849e19d77ea65aaa2188d12c20cee987ab6f321e002
+size 992877600

vibeasr-vae-encoder-i8_s.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4941c82608c253ec066b5cc74d3dd11a5c8fef96cccbc5b87359ef0fe4338df6
+size 703080064