---
license: apache-2.0
base_model: OpenMOSS-Team/MOSS-Audio-8B-Instruct
tags:
  - mlx
  - audio
  - moss-audio
  - asr
  - int4
  - apple-silicon
language:
  - en
  - zh
pipeline_tag: audio-text-to-text
library_name: mlx
---

# MOSS-Audio-8B-Instruct-MLX (hybrid: INT4 LLM + BF16 audio)

An Apple MLX conversion of **MOSS-Audio-8B-Instruct** — the ASR-strongest MOSS-Audio
checkpoint — for fast, low-memory inference on Apple Silicon. LLM quantized to uniform
INT4 (group_size 64); audio encoder + adapter + DeepStack kept in BF16.

> **Why this exists.** The community had MLX builds only of the *Thinking* variant.
> But Thinking is not ASR-optimized: under identical INT4 quantization it mis-spells
> letter-spoken tickers (e.g. "CRWD" → "CWD") and is unstable. **Instruct** transcribes
> them correctly. This build brings Instruct's transcription quality to MLX speed/memory.

中文：這是 **MOSS-Audio-8B-Instruct** 的 Apple MLX 轉換版（LLM uniform INT4 + audio 路徑 BF16）。
社群原本只有 *Thinking* 變體的 MLX 版，但 Thinking 非 ASR 優化——相同 INT4 量化下會把唸出字母的
ticker（如 "CRWD"）辨識成 "CWD" 且不穩定。**Instruct** 辨識正確。本版把 Instruct 的轉錄品質
帶到 MLX 的速度與記憶體。

## Measured (Apple M1 Max 32GB, 28s zh+en clip)

| Metric | PyTorch Instruct | **This (Instruct-MLX)** | Thinking-MLX |
|---|:---:|:---:|:---:|
| Ticker "CRWD" | C R W D ✅ | **C R W D ✅** | CWD ❌ |
| English term (TradingView) | ✅ | ✅ | ✅(loops) |
| Numerals | Chinese chars | **Arabic 47%** | Arabic |
| Speed | 1.8x realtime | **6–9x** | 5–8x |
| Peak memory | ~17 GB | **7.85 GB** | 7.85 GB |
| Disk | 18 GB | **5.9 GB** | 5.9 GB |

**Key finding.** Ticker-ASR degradation in the Thinking-MLX builds comes from the
Thinking/Instruct *training difference*, not from INT4 quantization — under the same
uniform INT4, Instruct keeps the ticker. So uniform 4-bit suffices; no mixed-precision needed.

## Usage

```bash
pip install mlx mlx-lm soundfile numpy
python inference.py --audio your_clip_16k_mono.wav
```

Transcription with per-segment timestamps (a Traditional-Chinese prompt triggers
zh-Hant output):

```bash
python run_moss.py --model . --audio clip.wav \
  --prompt "請逐句轉錄這段音訊，每句標註開始時間。" --temp 0 --repetition-penalty 1.02
```

- **Audio**: 16 kHz mono. Encoder window is Whisper-style **30 s max** — chunk longer audio.
- **Decoding**: use **greedy (temp=0)** for ASR fidelity. `temp>0` removes the rare
  tail digit-loop but degrades content (wrong numerals, out-of-order timestamps).
- **digit-loop**: occasionally the model fails to emit EOS and repeats a digit token
  at the very tail; post-truncate repeated trailing digits. Quantization weakens EOS;
  it is a known, harmless tail artifact for transcription use.

## How it was converted

Pure metadata-mapped weight conversion (no retraining):

1. `stage1_mapping.py` — verify every MLX target key is sourceable from the PyTorch
   checkpoint; discover the conv layout transform `transpose(0,2,3,1)`
   (PyTorch `[out,in,h,w]` → MLX `[out,h,w,in]`).
2. `stage2_convert.py` — extract `language_model.*` + `lm_head`, quantize to INT4
   (group_size 64) via mlx; extract audio encoder/adapter/DeepStack, apply the conv
   transpose, save BF16. Output mirrors the RumiLabs bridge layout exactly.

## Limitations

- 30-second audio window (chunk + offset timestamps for longer input).
- Tail digit-loop under greedy (post-truncate).
- Homophone errors on domain terms (e.g. 300均 → 三百軍) — fix with a glossary/post-pass.

## Credits

- Base model: [OpenMOSS-Team/MOSS-Audio](https://github.com/OpenMOSS/MOSS-Audio) (Apache-2.0)
- MLX bridge (encoder/mel/DeepStack port): [RumiLabs](https://huggingface.co/RumiLabs) Thinking-MLX builds
- Instruct→MLX conversion: this work

## License

Apache-2.0 (inherited from base model).