| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| base_model: Edge0/Audio8-ASR-Infinite |
| pipeline_tag: automatic-speech-recognition |
| library_name: onnxruntime |
| tags: |
| - onnx |
| - streaming |
| - speech-recognition |
| - asr |
| - voxtral |
| - qwen2.5 |
| - winstt |
| --- |
| |
| # Audio8-ASR-Infinite — ONNX (streaming, constant memory) |
|
|
| An ONNX Runtime export of **[Edge0/Audio8-ASR-Infinite](https://huggingface.co/Edge0/Audio8-ASR-Infinite)** |
| (revision `7476824bc222e4ad509d286e8cae8b8d3f371129`). All credit for the model, its training and |
| its streaming design goes to **Edge0**. This repository only re-packages the original weights as |
| split ONNX graphs with explicit state so the model can run natively streaming outside PyTorch. It |
| was made for [WinSTT](https://github.com/dahshury/WinSTT), but the graphs are plain ONNX and the IO |
| contract is documented below. |
|
|
| Licence: **Apache-2.0**, inherited from the original model. See `LICENSE`. |
|
|
| ## What the model is |
|
|
| - A **causal Voxtral-Realtime-style audio tower** with 32 layers, hidden size 1280, 32 heads × 64, |
| and sliding-window attention over 750 frames. The log-mel and the causal conv front end are |
| inside the graph. |
| - A projector into a **Qwen2.5-3B decoder** with 36 layers, 16 query heads and 2 KV heads, |
| head_dim 128, and a tied LM head. Voxtral's AdaRMSNorm delay modulation is applied per layer. |
| - **Semantic-VAD heads** on the decoder's last hidden state: 4 horizons (0.5 / 1 / 2 / 3 s) × 8 |
| classes. Class 0 at the 2 s horizon is the end-of-turn probability. |
| - About **4.09B parameters** in total. Languages: Chinese and English, selected by a prompt token. |
| - **Streaming clock.** One text token per 1280 samples (80 ms), with a fixed **480 ms** delay |
| (6 tokens). Text appears word by word while audio is still arriving. Audio length is unbounded: |
| the encoder keeps a 749-slot ring buffer and the decoder keeps a 375-slot rolling cache. When |
| the decoder cache fills, the oldest 38 entries are dropped after a 16-token stable prefix. |
| Memory therefore stays constant however long the input is. |
| |
| ## Files |
| |
| | File | Purpose | |
| |---|---| |
| | `audio_encoder_{int4,int8,fp16}.onnx` (+ `.onnx.data`) | mel + causal conv + 32-layer tower + projector, one call per batch of new 80 ms windows | |
| | `decoder_{int4,int8,fp16}.onnx` (+ `.onnx.data`) | Qwen2.5-3B step with KV cache in/out, last-position logits + VAD logits | |
| | `embed_tokens.bf16` | raw bf16 `[151936, 2048]` token-embedding table (looked up on the host) | |
| | `ada_scale.f32` | precomputed `(1 + ada_rms_norm(t_cond))` table `[8, 36, 2048]`, one row per (frame_len, delay) combo listed in `runtime.json` | |
| | `runtime.json` | format `winstt-audio8-infinite-v1`: streaming geometry, cache sizes, special tokens, VAD horizons, table layouts | |
| | `tokenizer.json`, `tokenizer_config.json`, `config.json` | from the original repo | |
| | `scripts/` | the export, graph builder, PyTorch reference and parity harness used to make this repo | |
|
|
| ### Download sizes per precision |
|
|
| Each size includes the shared tables and tokenizer (636 MB). |
|
|
| | Precision | Encoder | Decoder | Total | |
| |---|---|---|---| |
| | `int4` (MatMulNBits 4-bit, block 32, asymmetric, accuracy_level 4) | 0.66 GB | 1.98 GB | **3.27 GB** | |
| | `int8` (MatMulNBits 8-bit, block 32, accuracy_level 4) | 1.17 GB | 3.57 GB | **5.37 GB** | |
| | `fp16` (whole graph fp16; RMSNorm, softmax and mel in fp32; fp16 KV caches) | 1.99 GB | 6.17 GB | **8.80 GB** | |
|
|
| In every tier, only the linear weights are quantized; activations stay in fp32 (fp16 for the |
| `fp16` tier). |
|
|
| ## Graph IO contract |
|
|
| The rotary embedding is applied **on the host**. Keys are cached already rotated, using angles |
| computed in f64 and reduced mod 2π. Because of that, no cache ever needs re-rotation, except for |
| the decoder's rolling trim, where the surviving suffix is re-based by −38 positions. |
|
|
| **`audio_encoder_*.onnx`** |
| |
| | Name | Type / shape | Meaning | |
| |---|---|---| |
| | in `audio` | f32 `[B, S]` | B raw 16 kHz windows of equal length (one per new token, with 52.5 ms look-back / 2.5 ms look-ahead) | |
| | in `frame_len` | i64 `[1]` | encoder frames per text token (4 = 80 ms) | |
| | in `cos`, `sin` | f32 `[T, 64]` | RoPE factors of the T = B·4 new frames | |
| | in `attn_bias` | f32 `[T, 749 + T]` | additive mask (0 / −1e9): ring slots, then the new frames | |
| | in `past_key_i`, `past_value_i` | `[1, 32, 749, 64]` ×32 layers | encoder ring (rotated keys) | |
| | out `audio_embeds` | f32 `[1, T/4, 2048]` | one audio embedding per text token | |
| | out `key_i`, `value_i` | `[1, 32, T, 64]` | new keys and values to write into the ring | |
| |
| **`decoder_*.onnx`** |
| |
| | Name | Type / shape | Meaning | |
| |---|---|---| |
| | in `inputs_embeds` | f32 `[1, n, 2048]` | `embed_tokens[prev_token] + audio_embeds[k]` | |
| | in `ada_scale` | f32 `[36, 2048]` | the row of `ada_scale.f32` for the session's (frame_len, delay) | |
| | in `cos`, `sin` | f32 `[n, 128]` | RoPE factors of the new positions | |
| | in `attn_bias` | f32 `[n, 375 + n]` | additive mask: cache slots, then causal over the new positions | |
| | in `past_key_i`, `past_value_i` | `[1, 2, 375, 128]` ×36 layers | rolling decoder cache (fp16 in the `fp16` tier) | |
| | out `logits` | f32 `[1, 151936]` | last position | |
| | out `vad_logits` | f32 `[1, 4, 8]` | semantic-VAD heads, last position | |
| | out `key_i`, `value_i` | `[1, 2, n, 128]` | new cache entries | |
| |
| **Decode loop.** It follows the original `simulated_streaming_greedy`. |
| |
| 1. Window 0 covers audio `[0, 25·1280 + 40)`. Pass it to the encoder with prefill 25. |
| 2. The decoder prompt is `[BOS, <lang>, STREAMING_PAD × 23]`, added to the 25 audio embeddings. |
| 3. Every later 80 ms window yields one audio embedding and exactly one decoder step. The step's |
| input is the previously emitted token's embedding plus that audio embedding. |
| 4. Pick the next token by greedy argmax, with EOS masked out. |
| |
| The audio is left-padded by 18 tokens. At end of stream, add (6 + 1 + 10) × 1280 samples of |
| silence. Visible text is every token below id 151643, detokenized with byte-level BPE. |
| `scripts/audio8_infinite_ref.py` (class `Session`) is a complete NumPy reference of this loop. |
| |
| ## Parity and accuracy |
| |
| ### Setup |
| |
| - **Reference**: the original PyTorch weights in fp32. Decoding is exact greedy with EOS masked, |
| following the same streaming schedule. The checkpoint is read directly by |
| `scripts/audio8_infinite_ref.py`, so `transformers` is not involved. |
| - **Logit and token parity**: measured on JFK (11 s, 148 decoder steps) and on JFK ×6 (66 s, 835 |
| steps). The 66 s clip overflows the 375-slot decoder cache, so it goes through 13 rolling trims. |
| - **Accuracy**: measured with the WinSTT Rust runtime on two concatenated sets, each streamed as |
| one continuous recording of about 5 minutes: |
| - 40 LibriSpeech test-clean utterances (308 s), scored by WER; |
| - 25 FLEURS cmn_hans_cn utterances (290 s), scored by CER. |
| - **Scoring normalization**: English is lower-cased with punctuation stripped. Chinese is |
| NFKC-normalized with punctuation and whitespace stripped; digits and Chinese numerals are not |
| unified, which penalizes all models equally. |
| |
| ### Parity against PyTorch fp32 (JFK, 148 steps) |
| |
| | Precision | Max \|Δlogit\| | Mean per-step max \|Δlogit\| | Greedy tokens identical | Transcript identical | |
| |---|---|---|---|---| |
| | fp32 graph (reduced-size test config) | 2.6e-6 | n/a | yes | yes | |
| | `fp16` | 0.047 | 0.014 | **yes, 148/148** | yes | |
| | `int8` | 2.83 | 0.87 | **yes, 148/148** | yes | |
| | `int4` | 4.90 (before the first divergence) | 2.81 | no: 97.97% agree, first split at step 46 on a `[STREAMING_PAD]` vs `[STREAMING_WORD]` boundary marker | yes | |
| | legacy `int8dq` (not shipped) | n/a | n/a | no: split at step 7 | **no** ("For my fellow Americans, ask not. What…") | |
|
|
| The peak logit magnitude is about 37. |
|
|
| **Long audio (66 s, 835 steps, 13 rolling-cache trims):** |
| - The `int4` sequence of visible text tokens is **identical** to the PyTorch sequence (156/156 |
| tokens), and the transcripts are identical. |
| - The only differences are in the timing of the non-visible pad/word markers. |
|
|
| ### Accuracy (WinSTT Rust runtime, CPU, continuous streaming of the concatenated sets) |
|
|
| | Model | Precision | LibriSpeech test-clean (40 utts) WER | FLEURS zh (25 utts) CER | |
| |---|---|---|---| |
| | **Audio8-ASR-Infinite** | `int8` | **4.27%** | **11.25%** | |
| | **Audio8-ASR-Infinite** | `int4` | 4.66% | 11.85% | |
| | **Audio8-ASR-Infinite** | `fp16` | not run (too slow on CPU) | 11.37% | |
| | Audio8-ASR-0.1B (for comparison, chunked offline) | fp32 | 3.75% | 12.09% | |
| | ARK-ASR-0.6B (for comparison, chunked offline) | int8 | 2.72% | 11.00% | |
|
|
| Notes on the comparison rows: |
| - Audio8-ASR-0.1B and ARK-ASR-0.6B are offline models. WinSTT runs them on VAD-split chunks of |
| up to 24 s with full right context. |
| - Audio8-ASR-Infinite decodes the same audio strictly causally, with 480 ms of look-ahead. |
| - These are small subsets, so treat differences under about 1 point as noise. |
|
|
| **Set the language token.** Prompting with the wrong language makes the model *translate*. With |
| the English token, Chinese speech came out as English. At `int8` this also degenerated into a |
| repetition loop. The model cannot auto-detect the language, so pass `zh` or `en` explicitly. |
|
|
| ### Speed |
|
|
| All timings below were taken on a Windows desktop, a 24-thread CPU, while other heavy jobs were |
| running. Use them as relative numbers, not absolute ones. |
|
|
| - **CPU, 11 s clip, same moment** (MatMulNBits `int4`): RTF **2.1** on CPU vs **10.8** with |
| DirectML (RTX 3080 Ti). The transcripts were identical. |
| - **DirectML.** The streaming caches are re-fed from the host on every call: about 390 MB of |
| encoder ring per window and about 27 MB of decoder cache per token. That makes DirectML |
| roughly 5× slower than CPU here, so WinSTT pins this model to CPU. |
| - **CPU across precisions.** RTF ranged from 1.0 to 4.2 for `int4` / `int8` over the 5-minute |
| sets, depending on load. The two tiers were within the load noise of each other, so `int8` is |
| the recommended tier: it is more accurate and not measurably slower. |
| - **fp16** is slow on CPU: RTF 20.6 on the 290 s FLEURS set. It is included for GPU runtimes; |
| WinSTT itself offers only `int8` and `int4`. |
| - **Load and memory.** Load takes about 15 s. Peak RSS is about 5.4 GB at `int4`, and memory |
| stays constant regardless of audio length. |
|
|
| ## Reproducing |
|
|
| ```bash |
| python scripts/audio8_infinite_export.py --ckpt <snapshot of Edge0/Audio8-ASR-Infinite> --out bundle --precision int4 int8 fp16 |
| A8I_CKPT=<snapshot> A8I_BUNDLE=bundle python scripts/audio8_infinite_parity.py torch --sets jfk |
| A8I_CKPT=<snapshot> A8I_BUNDLE=bundle python scripts/audio8_infinite_parity.py onnx --prec int4 --sets jfk ls_clean fleurs_zh --keep-logits |
| python scripts/audio8_infinite_parity.py report |
| ``` |
|
|
| The scripts need `onnx`, `onnxruntime` (≥ 1.20 for 8-bit MatMulNBits), `numpy`, and, for the parity harness, |
| `tokenizers`, `soundfile` and `torch`. The checkpoint is read straight from the safetensors |
| files, so no `transformers` install is needed. They were built with onnx 1.23.1, onnxruntime 1.30.0 |
| and torch 2.13.0. |
|
|
| ## Credits |
|
|
| - Model, training and streaming design: **Edge0**, |
| [Edge0/Audio8-ASR-Infinite](https://huggingface.co/Edge0/Audio8-ASR-Infinite), Apache-2.0. |
| - The architecture builds on Mistral's Voxtral-Realtime audio tower and Alibaba's Qwen2.5-3B. |
| - ONNX export and streaming runtime: WinSTT. |
|
|
