| --- |
| license: apache-2.0 |
| language: |
| - zh |
| - en |
| pipeline_tag: audio-text-to-text |
| tags: |
| - audio |
| - speech-recognition |
| - audio-understanding |
| - audio-qa |
| - transcription |
| - ggml |
| - gguf |
| - moss-audio |
| - qwen3 |
| - deepstack |
| library_name: ggml |
| base_model: OpenMOSS-Team/MOSS-Audio-4B-Instruct |
| --- |
| |
| # MOSS-Audio-4B-Instruct -- GGUF (ggml-quantised) |
|
|
| GGUF / ggml conversions of [`OpenMOSS-Team/MOSS-Audio-4B-Instruct`](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-4B-Instruct) for use with `crispasr --backend moss-audio` from **[CrispStrobe/CrispASR](https://github.com/CrispStrobe/CrispASR)**. |
|
|
| MOSS-Audio-4B-Instruct is OpenMOSS's ~4.6 B parameter **audio-understanding** model: |
|
|
| - **First audio-understanding backend in CrispASR** -- not just ASR but also audio QA, scene description, music analysis, meeting summarisation |
| - **Mandarin + English** speech recognition and audio understanding |
| - **DeepStack cross-layer feature injection** -- multi-resolution encoder taps at layers 8/16/24 injected into the LM's early layers for fine-grained prosody + semantic awareness |
| - **Time-aware ASR** with explicit time-marker tokens for word-level and sentence-level timestamps |
| - **Apache-2.0** licence |
|
|
| ## Files |
|
|
| | File | Size | Notes | |
| | --- | ---: | --- | |
| | `moss-audio-4b-instruct-f16.gguf` | 9.73 GB | F16, full precision | |
| | `moss-audio-4b-instruct-q4_k.gguf` | 2.75 GB | **Q4_K -- recommended default** | |
| |
| ## Quick Start |
| |
| ```bash |
| # 1. Build the runtime |
| git clone https://github.com/CrispStrobe/CrispASR |
| cd CrispASR |
| cmake -G Ninja -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=OFF |
| cmake --build build -j$(nproc) --target crispasr-cli |
| |
| # 2. Download a quantisation |
| huggingface-cli download cstr/MOSS-Audio-4B-Instruct-GGUF \ |
| moss-audio-4b-instruct-q4_k.gguf --local-dir . |
| |
| # 3. Transcribe audio |
| ./build/bin/crispasr \ |
| -m moss-audio-4b-instruct-q4_k.gguf \ |
| -f your-audio.wav \ |
| --backend moss-audio -t 4 |
| |
| # 4. Audio understanding (custom prompt) |
| ./build/bin/crispasr \ |
| -m moss-audio-4b-instruct-q4_k.gguf \ |
| -f your-audio.wav \ |
| --backend moss-audio \ |
| --prompt "Describe the sounds in this audio clip." |
| ``` |
| |
| |
| ## Verified end-to-end output |
| |
| **JFK sample (samples/jfk.wav, 11s):** |
| > And so, my fellow Americans, ask not what your country can do for you, ask what you can do for your country. |
| |
| Verified on Q4_K (3.8 GB, F16 encoder + Q4_K LLM). All 6 crispasr-diff stages PASS at cos >= 0.999. |
| |
| ## Architecture |
| |
| | Component | Details | |
| | --- | --- | |
| | Audio encoder | 32-layer Whisper-style transformer (d=1280, 20 heads, head_dim=64, FFN=5120, GELU, LayerNorm, eps=1e-5) | |
| | Conv stem | 3x Conv2d(stride=2, channels=480, kernel=3x3, pad=1) -> 8x temporal downsample (128 mel bins -> 16 freq bins) | |
| | Stem projection | Linear(480x16=7680 -> 1280) + sinusoidal positional embedding (max 1500 positions) | |
| | DeepStack taps | Encoder layers [8, 16, 24] -> 3 independent GatedMLP(1280 -> 8192 -> 2560, SiLU) | |
| | DeepStack injection | Residual add at LM layers [0, 1, 2] at audio-token positions | |
| | Audio adapter | GatedMLP(1280 -> 8192 -> 2560, SiLU) for final encoder output | |
| | LM backbone | 36-layer Qwen3 (hidden=2560, 32 Q-heads / 8 KV-heads, head_dim=128, QK-norm, SwiGLU FFN=9728, RoPE theta=1M) | |
| | Output head | Linear(2560 -> 151936), untied from embedding | |
| | Vocab | 151936 Qwen3 BPE (151643 regular + 293 special tokens) | |
| | Audio input | 16 kHz mono, 128 mel bins, n_fft=400, hop=160 | |
| | Audio tokens | 12.5 Hz after 8x conv downsample, time markers every 2 seconds | |
| | Parameters | ~4.6 B total (encoder ~650M + adapter/deepstack ~120M + LM ~3.8B) | |
| |
| ### Special tokens |
| |
| | Token | ID | Purpose | |
| | --- | --- | --- | |
| | `<\|AUDIO\|>` | 151654 | Audio frame placeholder (replaced by encoder embeddings) | |
| | `<\|audio_bos\|>` | 151669 | Audio segment start marker | |
| | `<\|audio_eos\|>` | 151670 | Audio segment end marker | |
| | `<\|im_start\|>` | 151644 | Chat turn start | |
| | `<\|im_end\|>` | 151645 | Chat turn end / EOS | |
| |
| ## How this was made |
| |
| 1. **Inspect** the HF safetensors: 3 shards, 901 tensors total -- audio encoder (conv stem + 32 transformer layers + layer_norm), audio adapter (1 GatedMLP), deepstack mergers (3 GatedMLPs), language model (embedding + 36 Qwen3 layers + final norm + lm_head). |
|
|
| 2. **Convert** with [`models/convert-moss-audio-to-gguf.py`](https://github.com/CrispStrobe/CrispASR/blob/feature/moss-audio/models/convert-moss-audio-to-gguf.py): stream BF16 tensors one-at-a-time via `safe_open`, remap HF tensor names (`audio_encoder.layers.N.self_attn.q_proj` -> `enc.blk.N.attn.q`, `deepstack_audio_merger_list.N.gate_proj` -> `deepstack.N.gate`, `language_model.layers.N.mlp.gate_proj` -> `llm.blk.N.ffn.gate`, etc.), write F16 + F32 (norms/biases). BPE vocab + merges from `vocab.json` + `merges.txt` + `added_tokens.json`. |
|
|
| 3. **Quantize** with `crispasr-quantize`: F16 -> Q4_K (2D+ tensors quantised, 1D biases/norms kept F32). |
| |
| 4. **C++ runtime** in [`src/moss_audio.{h,cpp}`](https://github.com/CrispStrobe/CrispASR/blob/feature/moss-audio/src/moss_audio.cpp): GGUF mmap, encoder graph (conv stem + 32 WhisperEncoderLayers with bidirectional attention + DeepStack tap capture), adapter/merger GatedMLP graphs, per-layer DeepStack injection into LM via pre-scattered residuals, KV-cached Qwen3 decode with `core_attn::kv_self_attn` (QK-norm, RoPE, GQA), greedy decode with chat-template prompt builder. |
|
|
| ## Upstream |
|
|
| - Model: [`OpenMOSS-Team/MOSS-Audio-4B-Instruct`](https://huggingface.co/OpenMOSS-Team/MOSS-Audio-4B-Instruct) (Apache-2.0) |
| - Code: [`OpenMOSS/MOSS-Audio`](https://github.com/OpenMOSS/MOSS-Audio) |
| - Runtime: [`CrispStrobe/CrispASR`](https://github.com/CrispStrobe/CrispASR) branch `feature/moss-audio` |
|
|
