README.md

---
license: apache-2.0
language:
- en
- zh
tags:
- moss
- audio
- speech
- asr
- diarization
- timestamp-asr
- long-form-audio
- multimodal
pipeline_tag: audio-text-to-text
---

# MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize is an open-source **speech transcription and diarization model** from the OpenMOSS team. It performs unified modeling over long-form, multi-speaker audio, supporting **automatic speech recognition, speaker-aware transcription, speaker diarization, timestamp prediction, and compact transcript generation**.

Given an audio or video file, MOSS-Transcribe-Diarize generates a speaker-aware transcript in one pass, with segment timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and `[S03]`.

## News

* 2026.05.21: We have released MOSS-Transcribe-Diarize.

## Contents

- [Introduction](#introduction)
- [Model Architecture](#model-architecture)
- [Released Models](#released-models)
- [Evaluation](#evaluation)
- [Quickstart](#quickstart)
  - [Environment Setup](#environment-setup)
  - [Command Line Inference](#command-line-inference)
  - [Python Usage](#python-usage)
- [Output Format](#output-format)
- [More Information](#more-information)
- [LICENSE](#license)
- [Citation](#citation)

## Introduction

Long-form speech understanding requires more than plain transcription. For meetings, calls, podcasts, interviews, lectures, videos, and other real-world recordings, a useful transcript should identify **what was said**, **who said it**, and **when it was said**.

MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model.

* **Long-form ASR**: Transcribes long audio and video recordings into text.
* **Speaker-aware transcription**: Adds anonymous speaker labels to each speech segment.
* **Speaker diarization**: Produces "who spoke when" style output without a separate diarization pipeline.
* **Timestamp prediction**: Generates segment-level start and end timestamps.
* **Audio and video input**: Supports common audio files and video containers decoded through PyAV.
* **Promptable generation**: Allows users to customize the transcription instruction.

## Model Architecture

<p align="center">
  <img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize model architecture" width="900">
</p>

MOSS-Transcribe-Diarize follows a modular audio-language design comprising three components: an audio encoder, a modality adapter, and a causal language model. Raw audio is converted into log-mel features, encoded by a Whisper-style audio encoder, projected into the language model embedding space through an MLP adapter, and then consumed by a Qwen3-style causal decoder for auto-regressive text generation.

The model uses audio placeholder tokens in the text sequence. During the forward pass, projected audio representations replace the corresponding placeholder embeddings, allowing the language model to generate timestamped, speaker-aware transcripts conditioned on the input audio.

| Component | Specification |
| --- | --- |
| Text backbone | Qwen3-0.6B style causal decoder |
| Audio encoder | Whisper-Medium encoder configuration |
| Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
| Audio-text adapter | 4x temporal merge + MLP adapter |
| Fusion method | Audio features replace `<|audio_pad|>` embeddings |
| Output format | Compact `[start][Sxx]text[end]` transcript |

## Released Models

| Model | Audio Encoder | LLM Backbone | Hugging Face |
| --- | --- | --- | --- |
| **MOSS-Transcribe-Diarize** | Whisper-style audio encoder | Qwen3-0.6B style decoder | [Hugging Face](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) |

> More model variants may be released in the future. Stay tuned!

## Evaluation

We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates unavailable results.

| Dataset | Metric | Doubao | ElevenLabs | GPT-4o | Gemini 2.5 Pro | Gemini 3 Pro | VIBEVOICE ASR | MOSS Transcribe Diarize |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **AISHELL-4** | CER down | 18.18 | 19.58 | - | 42.70 | 22.75 | 21.40 | **14.19** |
| | cpCER down | 27.86 | 37.95 | - | 53.42 | 27.43 | 24.99 | **14.98** |
| | Delta-cp down | 9.68 | 18.36 | - | 10.72 | 4.68 | 3.59 | **0.79** |
| **Podcast** | CER down | 7.93 | 8.50 | - | 7.38 | - | 27.94 | **4.46** |
| | cpCER down | 10.54 | 11.34 | - | 10.23 | - | 48.30 | **6.97** |
| | Delta-cp down | 2.61 | 2.85 | - | 2.85 | - | 20.36 | **2.50** |
| **Movies** | CER down | 9.94 | 11.49 | 14.37 | 15.46 | 8.62 | 14.59 | **6.58** |
| | cpCER down | 30.88 | 17.85 | 23.67 | 24.15 | 14.73 | 42.54 | **13.68** |
| | Delta-cp down | 20.94 | **6.37** | 9.31 | 8.69 | 6.11 | 27.94 | 7.24 |
| **Alimeeting** | CER down | 25.25 | 25.70 | - | 27.43 | 26.75 | 27.40 | **24.80** |
| | cpCER down | 37.57 | 36.69 | - | 41.64 | 32.84 | 29.33 | **21.51** |
| | Delta-cp down | 12.31 | 10.99 | - | 14.21 | 6.09 | 1.93 | **-0.33** |

## Quickstart

### Environment Setup

We recommend Python 3.12 with a clean Conda environment. This repository uses custom Transformers model and processor code, so always load the model and processor with `trust_remote_code=True`.

```bash
git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
cd MOSS-Transcribe-Diarize

conda create -n moss-transcribe-diarize python=3.12 -y
conda activate moss-transcribe-diarize

conda install -c conda-forge "ffmpeg=7" -y
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
```

Optional: if your GPU supports FlashAttention 2, install the optional runtime with:

```bash
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
```

### Command Line Inference

Run greedy decoding:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding greedy \
  --max-new-tokens 2048
```

Run sampling decoding:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding sample \
  --temperature 0.7 \
  --max-new-tokens 2048
```

Return JSON output:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --json
```

Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.

### Python Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "/path/to/audio_or_video.mp4"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=True,
)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result["text"])
```

The default prompt asks the model to output each speech segment with a start timestamp, a speaker label, transcript text, and an end timestamp. You can customize the instruction with:

```python
messages = build_transcription_messages(
    audio_path,
    prompt="Please transcribe the audio with timestamps and speaker labels.",
)
```

The same can be done from the command line with `--prompt`.

## Output Format

The canonical output format is:

```text
[start_time][Sxx]transcribed speech[end_time]
```

Example:

```text
[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
```

In this format:

* `start_time` and `end_time` are timestamps in seconds.
* `[S01]`, `[S02]`, and similar labels are anonymous model-generated speaker labels.
* Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.

## More Information

* **MOSI.AI**: <https://mosi.cn>
* **OpenMOSS**: <https://www.open-moss.com>

## LICENSE

MOSS-Transcribe-Diarize is licensed under the Apache License 2.0.

## Citation

```bibtex
@misc{mosstranscribediarize2026,
      title={MOSS-Transcribe-Diarize},
      author={OpenMOSS Team},
      year={2026},
      howpublished={\url{https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize}},
      note={Hugging Face model repository}
}
```