--- license: apache-2.0 library_name: transformers language: - en - zh tags: - moss - audio - speech - asr - diarization - timestamp-asr - long-form-audio - multimodal - custom_code pipeline_tag: audio-text-to-text --- # MOSS-Transcribe-Diarize
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness. Given an audio or video file, the model generates a compact speaker-aware transcript in one pass, including timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and beyond. ## News * 2026-07-09: Released MOSS-Transcribe-Diarize 0.9B. ## Contents - [Introduction](#introduction) - [Model Architecture](#model-architecture) - [Evaluation](#evaluation) - [Quickstart](#quickstart) - [Serve with SGLang Omni](#serve-with-sglang-omni) - [Serving with Hugging Face](#serving-with-native-hugging-face-transformers) - [Python Usage](#python-usage) - [Custom Prompt and Hotwords](#custom-prompt-and-hotwords) - [Serve with vLLM](#serve-with-vllm) - [Subtitle Web App](#subtitle-web-app) - [Output Format](#output-format) - [More Information](#more-information) - [License](#license) - [Citation](#citation) ## Introduction MOSS-Transcribe-Diarize 0.9B turns real-world long-form audio into structured, speaker-aware transcripts in one pass. Instead of stitching together separate ASR and diarization systems, it jointly performs speech transcription and speaker diarization, producing time-aligned text with consistent speaker labels. The model is built for meetings, calls, podcasts, interviews, lectures, videos, and other long or messy multi-speaker recordings. It can also emit acoustic event annotations, giving downstream systems a richer view of what happened, who spoke, and when. Core capabilities: * **Long-form transcription**: Converts long audio or video recordings into timestamped text. * **Speaker-aware diarization**: Assigns anonymous speaker labels such as `[S01]` and `[S02]` without a separate diarization pipeline. * **Promptable generation**: Supports custom transcription instructions, hotwords, and acoustic event annotations. ## Model Architecture

MOSS-Transcribe-Diarize 0.9B model architecture

| Component | Specification | |---|---| | Text backbone | Qwen3-0.6B style causal decoder | | Audio encoder | Whisper-Medium encoder configuration | | Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks | | Audio-text bridge | 4x temporal merge + MLP adaptor | | Fusion | Audio features replace <|audio_pad|> embeddings via `masked_scatter` | | Output format | Compact `[start][Sxx]text[end]` transcript with speaker tags such as `[S01]` | This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`. ## Evaluation We evaluate MOSS-Transcribe-Diarize using three objective metrics: Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates that the result is unavailable.
Model AISHELL‑4 Alimeeting Podcast Movies
CER↓cpCER↓Δcp↓ CER↓cpCER↓Δcp↓ CER↓cpCER↓Δcp↓ CER↓cpCER↓Δcp↓
Doubao 18.1827.869.68 25.2537.5712.31 7.9310.542.61 9.9430.8820.94
ElevenLabs 19.5837.9518.36 25.7036.6910.99 8.5011.342.85 11.4917.856.37
GPT-4o --- --- --- 14.3723.679.31
Gemini 2.5 Pro 42.7053.4210.72 27.4341.6414.21 7.3810.232.85 15.4624.158.69
Gemini 3 Pro 22.7527.434.68 26.7532.846.09 --- 8.6214.736.11
VIBEVOICE ASR 21.4024.993.59 27.4029.331.93 27.9448.3020.36 14.5942.5427.94
MOSS Transcribe Diarize 0.9B 14.8415.830.99 24.8622.17-2.69 5.977.371.40 6.3612.766.40
MOSS Transcribe Diarize Pro 13.7814.020.24 18.2213.94-4.27 4.466.972.51 5.8611.785.92
## Quickstart ### Serve with SGLang Omni The recommended way to serve MOSS-Transcribe-Diarize is [SGLang Omni](https://github.com/sgl-project/sglang-omni) through the OpenAI-compatible `/v1/audio/transcriptions` endpoint. Install `sglang-omni` by following the [Installation guide](https://github.com/sgl-project/sglang-omni/blob/main/docs/get_started/installation.md), then download the model: ```bash hf download OpenMOSS-Team/MOSS-Transcribe-Diarize ``` Serve the model: ```bash sgl-omni serve \ --model-path OpenMOSS-Team/MOSS-Transcribe-Diarize \ --port 8000 \ --max-running-requests 16 \ --cuda-graph-max-bs 16 \ --mem-fraction-static 0.80 ``` Use `response_format=verbose_json` when you need parsed speaker segments. `json` returns the raw transcript text only. ```bash curl -X POST http://localhost:8000/v1/audio/transcriptions \ -F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \ -F file=@audio.wav \ -F response_format=verbose_json ``` ```python import requests with open("audio.wav", "rb") as f: resp = requests.post( "http://localhost:8000/v1/audio/transcriptions", data={ "model": "OpenMOSS-Team/MOSS-Transcribe-Diarize", "response_format": "verbose_json", }, files={"file": ("audio.wav", f, "audio/wav")}, timeout=300, ) resp.raise_for_status() payload = resp.json() print(payload["text"]) for segment in payload.get("segments", []): print( f"[{segment['start']:.2f}-{segment['end']:.2f}] {segment['text']}" ) ``` For longer multi-speaker audio, raise `max_new_tokens` so the decoder can finish the full diarized transcript: ```bash curl -X POST http://localhost:8000/v1/audio/transcriptions \ -F model=OpenMOSS-Team/MOSS-Transcribe-Diarize \ -F file=@audio.wav \ -F response_format=verbose_json \ -F max_new_tokens=65536 ``` | Parameter | Type | Default | Description | |---|---|---|---| | `file` | file | required | Audio file uploaded as multipart form data | | `model` | string | server default | Model identifier | | `language` | string | unset | Optional language hint | | `response_format` | string | `json` | `json`, `verbose_json`, or `text` | | `temperature` | float | model default (`0.0`) | Sampling temperature | | `max_new_tokens` | int | `5120` | Max generated tokens; raise for long audio (e.g. `65536`) | | `prompt` | string | unset | Optional instruction override; omit to use the built-in transcribe+diarize prompt | For benchmarking, performance numbers, and more details, see the [SGLang Omni cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/moss_transcribe_diarize.md), here we list the performance of short/long-sequence mutli-speaker ASR tasks on single H100: 1. `movies800times` for short sequence ASR: | Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s | |---:|---:|---:|---:|---:| | 1 | 2.57 | 0.388 | 0.0612 | 29.76 | | 2 | 4.89 | 0.409 | 0.0659 | 56.55 | | 4 | 6.62 | 0.513 | 0.0790 | 76.64 | | 8 | 6.80 | 0.533 | 0.0810 | 78.70 | | 16 | 7.08 | 0.659 | 0.0922 | 81.98 | 2. `aishell4_long` for long sequence ASR: | Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s | |---:|---:|---:|---:|---:| | 1 | 0.022 | 45.2 | 0.0197 | 50.64 | | 2 | 0.032 | 60.7 | 0.0265 | 74.25 | | 4 | 0.036 | 105.6 | 0.0461 | 81.64 | | 8 | 0.040 | 172.6 | 0.0754 | 90.62 | | 16 | 0.043 | 282.8 | 0.1237 | 98.83 | ### Serving with Native Hugging Face Transformers Use a clean Python environment. The model uses custom Transformers code, so load the model and processor with `trust_remote_code=True`. ```bash conda create -n moss-transcribe-diarize python=3.12 -y conda activate moss-transcribe-diarize git clone https://github.com/OpenMOSS/MOSS-Transcribe-Diarize.git cd MOSS-Transcribe-Diarize pip install --index-url https://download.pytorch.org/whl/cu128 torch torchaudio pip install -e . ``` The GitHub package provides helper utilities such as audio/video loading, transcription message construction, transcript parsing, CLI inference, and the subtitle web app. The model weights and remote-code model files are loaded from this Hugging Face repository. ### Python Usage ```python import torch from transformers import AutoModelForCausalLM, AutoProcessor from moss_transcribe_diarize import parse_transcript from moss_transcribe_diarize.inference_utils import ( build_transcription_messages, generate_transcription, resolve_device, ) model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize" audio_path = "audio.wav" device = resolve_device("auto") dtype = torch.bfloat16 if device.type == "cuda" else torch.float32 model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype="auto", ).to(dtype=dtype).to(device).eval() processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True, ) messages = build_transcription_messages(audio_path) result = generate_transcription( model, processor, messages, max_new_tokens=2048, do_sample=False, device=device, dtype=dtype, ) print(result["text"]) for segment in parse_transcript(result["text"]): print(segment.start, segment.end, segment.speaker, segment.text) ``` The message flow follows the common Qwen multimodal pattern: 1. `processor.apply_chat_template(messages, tokenize=False)` renders text with audio placeholders. 2. The helper utilities load audio waveforms from the same messages. 3. `processor(text=text, audio=audios)` computes Whisper input features and expands audio placeholders. 4. `model.generate(...)` produces timestamped transcription and diarization text. ### Custom Prompt and Hotwords The default prompt is optimized for timestamped transcription and speaker diarization: ```text 请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。 ``` To add hotwords, append a short hint to the default prompt: ```text 请将音频转写为文本，每一段需以起始时间戳和说话人编号（[S01]、[S02]、[S03]…）开头，正文为对应的语音内容，并在段末标注结束时间戳，以清晰标明该段语音范围。热词提示：热词1, 热词2, 热词3 ``` More prompt recipes are available in the GitHub repository: ### Serve with vLLM MOSS-Transcribe-Diarize also supports vLLM serving through the OpenAI-compatible transcription API. Use a pinned vLLM nightly build that includes the MOSS-Transcribe-Diarize model registration. Choose one of the following commands: for CUDA 12 environments, use `cu129`; for CUDA 13 environments, use `cu130`. ```bash uv pip install -U vllm \ --torch-backend=auto \ --extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu129 ``` or: ```bash uv pip install -U vllm \ --torch-backend=auto \ --extra-index-url https://wheels.vllm.ai/68b4a1d582818e67adc903bf1b8fc5a5447da2fa/cu130 ``` ```bash vllm serve OpenMOSS-Team/MOSS-Transcribe-Diarize --trust-remote-code ``` ```bash curl http://localhost:8000/v1/audio/transcriptions \ -F model="OpenMOSS-Team/MOSS-Transcribe-Diarize" \ -F file=@"audio.wav" \ -F response_format="json" \ -F temperature="0" ``` ### Subtitle Web App The source package includes a local subtitle workflow for upload, review, subtitle export, and optional FFmpeg burn-in: ```bash mtd-subtitle-web \ --model OpenMOSS-Team/MOSS-Transcribe-Diarize \ --host 127.0.0.1 \ --port 7860 ``` Open `http://127.0.0.1:7860`, upload an audio/video file, review the parsed subtitle segments, then download JSON/SRT/ASS or burn an MP4 if `ffmpeg` and `ffprobe` are available on `PATH`. For batch processing: ```bash mtd-subtitle /path/to/input.mp4 \ --model OpenMOSS-Team/MOSS-Transcribe-Diarize \ --out-dir runs/example \ --render ``` ## Output Format The canonical output format is: ```text [start_time][Sxx]transcribed speech[end_time] ``` Example: ```text [0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76] ``` In this format: * `start_time` and `end_time` are timestamps in seconds. * `[S01]`, `[S02]`, and similar labels are anonymous model-generated speaker labels. * Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities. ## More Information * **GitHub**: * **MOSI.AI**: * **OpenMOSS**: ## License MOSS-Transcribe-Diarize 0.9B is licensed under the Apache License 2.0. ## Citation If you use MOSS-Transcribe-Diarize 0.9B, please cite the technical report: ```bibtex @misc{moss_transcribe_diarize_2026, title={MOSS Transcribe Diarize Technical Report}, author={{MOSI.AI}}, year={2026}, eprint={2601.01554}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2601.01554} } ```