README.md

---
license: apache-2.0
language:
- zh
- en
library_name: transformers
pipeline_tag: automatic-speech-recognition
tags:
- automatic-speech-recognition
- speaker-diarization
- timestamped-transcription
- long-form-audio
- audio-to-text
- multimodal
- transformers
- remote-code
---

# MOSS-Transcribe-Diarize

[English](#english) | [简体中文](#简体中文)

## English

**MOSS-Transcribe-Diarize** is an end-to-end audio understanding model for long-form, multi-speaker transcription. Given an audio or video file, it generates a compact speaker-aware transcript with timestamps, speaker labels, and transcribed text in one pass.

The model is designed for real-world recordings such as meetings, calls, podcasts, interviews, lectures, videos, and other long-form audio scenarios where multiple speakers and long context are common.

## Contents

- [Introduction](#introduction)
- [Model Architecture](#model-architecture)
- [Output Format](#output-format)
- [Intended Use](#intended-use)
- [Quickstart](#quickstart)
- [Evaluation](#evaluation)
- [Ethical Considerations](#ethical-considerations)
- [License](#license)
- [Citation](#citation)
- [简体中文](#简体中文)

## Introduction

Long-form speech transcription often requires more than plain ASR. In meetings, calls, podcasts, interviews, lectures, and video content, a useful transcript should answer three questions at the same time: what was said, who said it, and when it was said.

MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model:

- **Long-form ASR**: Transcribes long audio and video recordings into text.
- **Speaker-aware transcription**: Adds anonymous speaker labels such as `[S01]`, `[S02]`, and `[S03]`.
- **Timestamp prediction**: Produces segment-level start and end timestamps.
- **Audio/video input**: Supports common audio files and video containers decoded through PyAV.
- **Promptable generation**: Allows users to customize the transcription instruction.

## Model Architecture

MOSS-Transcribe-Diarize combines a Whisper-style audio encoder with a Qwen3-style causal language model decoder. Audio is converted into 16 kHz log-mel features, encoded by the audio encoder, merged temporally, projected through an MLP adaptor, and inserted into the language model embedding stream at audio placeholder positions.

| Component | Specification |
|---|---|
| Text backbone | Qwen3-0.6B style causal decoder |
| Audio encoder | Whisper-Medium encoder configuration |
| Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
| Audio-text bridge | 4x temporal merge + MLP adaptor |
| Fusion method | Audio features replace `<|audio_pad|>` embeddings |
| Output format | Compact `[start][Sxx]text[end]` transcript |

## Output Format

The default prompt asks the model to output each speech segment in the following format:

```text
[start_time][Sxx]transcribed speech[end_time]
```

Example:

```text
[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
```

In this format:

- `start_time` and `end_time` are timestamps in seconds.
- `[S01]`, `[S02]`, and similar labels are anonymous speaker labels generated by the model.
- Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.

## Intended Use

This model is intended for:

- Long-form automatic speech recognition.
- Speaker-aware transcription.
- Speaker diarization with anonymous speaker labels.
- Timestamped transcripts for meetings, podcasts, interviews, lectures, calls, and videos.
- Downstream audio understanding pipelines that need both transcript text and "who spoke when" information.

This model is not intended to be used as the sole source of truth for medical, legal, financial, safety-critical, or other high-stakes transcription without human review.

## Quickstart

### Environment Setup

This repository uses custom Transformers model and processor code. Always load the model and processor with `trust_remote_code=True`.

The code requires Python 3.10 or later and is tested with Transformers 5.x.

```bash
python -m pip install -U pip
python -m pip install "transformers>=5.0.0,<6.0.0" "torch>=2.8" "torchaudio>=2.8"
python -m pip install "safetensors>=0.6.2" "numpy>=1.26,<2" "av>=14.0" "librosa>=0.11.0" "soundfile>=0.12" "soxr>=0.5" packaging
```

For command line inference and helper utilities, clone the repository and install it in editable mode:

```bash
git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
cd MOSS-Transcribe-Diarize
python -m pip install -e .
```

### Command Line Inference

Run greedy decoding:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding greedy \
  --max-new-tokens 2048
```

Run sampling decoding:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding sample \
  --temperature 0.7 \
  --max-new-tokens 2048
```

Return JSON output:

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --json
```

Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.

### Python Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "/path/to/audio_or_video.mp4"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=True,
)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result["text"])
```

You can customize the transcription instruction by passing a custom prompt:

```python
messages = build_transcription_messages(
    audio_path,
    prompt="Please transcribe the audio with timestamps and speaker labels.",
)
```

The same can be done from the command line with `--prompt`.

## Evaluation

We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates unavailable results.

| Dataset | Metric | Doubao | ElevenLabs | GPT-4o | Gemini 2.5 Pro | Gemini 3 Pro | VIBEVOICE ASR | MOSS Transcribe Diarize |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **AISHELL-4** | CER ↓ | 18.18 | 19.58 | - | 42.70 | 22.75 | 21.40 | **14.19** |
| | cpCER ↓ | 27.86 | 37.95 | - | 53.42 | 27.43 | 24.99 | **14.98** |
| | Delta-cp ↓ | 9.68 | 18.36 | - | 10.72 | 4.68 | 3.59 | **0.79** |
| **Podcast** | CER ↓ | 7.93 | 8.50 | - | 7.38 | - | 27.94 | **4.46** |
| | cpCER ↓ | 10.54 | 11.34 | - | 10.23 | - | 48.30 | **6.97** |
| | Delta-cp ↓ | 2.61 | 2.85 | - | 2.85 | - | 20.36 | **2.50** |
| **Movies** | CER ↓ | 9.94 | 11.49 | 14.37 | 15.46 | 8.62 | 14.59 | **6.58** |
| | cpCER ↓ | 30.88 | 17.85 | 23.67 | 24.15 | 14.73 | 42.54 | **13.68** |
| | Delta-cp ↓ | 20.94 | **6.37** | 9.31 | 8.69 | 6.11 | 27.94 | 7.24 |
| **Alimeeting** | CER ↓ | 25.25 | 25.70 | - | 27.43 | 26.75 | 27.40 | **24.80** |
| | cpCER ↓ | 37.57 | 36.69 | - | 41.64 | 32.84 | 29.33 | **21.51** |
| | Delta-cp ↓ | 12.31 | 10.99 | - | 14.21 | 6.09 | 1.93 | **-0.33** |

## Ethical Considerations

Users should obtain appropriate consent before transcribing, diarizing, storing, or sharing recordings of people. The model generates anonymous speaker labels and does not verify real-world identities. Do not use the model for surveillance, unauthorized monitoring, or decisions that materially affect people without proper safeguards and human oversight.

## License

This project is released under the Apache License 2.0.

## Citation

Citation information will be added with the public model release.

---

## 简体中文

**MOSS-Transcribe-Diarize** 是一个面向长音频、多说话人场景的端到端音频理解模型。给定音频或视频文件后，模型可以在一次生成中输出带时间戳、说话人标签和转写文本的紧凑式说话人感知转录结果。

该模型适用于会议、电话、播客、访谈、课程、视频等真实录音场景，尤其适合需要同时处理长上下文和多说话人的音频内容。

## 目录

- [简介](#简介)
- [模型架构](#模型架构)
- [输出格式](#输出格式-1)
- [适用场景](#适用场景)
- [快速开始](#快速开始)
- [评测结果](#评测结果)
- [伦理说明](#伦理说明)
- [许可证](#许可证)
- [引用](#引用)
- [English](#english)

## 简介

长音频转写通常不只是普通 ASR。对于会议、电话、播客、访谈、课程和视频内容，一个有用的转录结果往往需要同时回答三个问题：说了什么、是谁说的、什么时候说的。

MOSS-Transcribe-Diarize 将这些能力统一在一个生成式模型中：

- **长音频语音识别**：将长音频和视频录音转写为文本。
- **说话人感知转写**：输出 `[S01]`、`[S02]`、`[S03]` 等匿名说话人标签。
- **时间戳预测**：为语音片段生成起止时间戳。
- **音频/视频输入**：支持常见音频文件和通过 PyAV 解码的视频容器。
- **可提示生成**：支持用户自定义转写指令。

## 模型架构

MOSS-Transcribe-Diarize 结合了 Whisper 风格的音频编码器和 Qwen3 风格的因果语言模型解码器。音频首先被转换为 16 kHz log-mel 特征，随后经过音频编码器、时间维度合并和 MLP 适配器投影，最终插入到语言模型中音频占位符对应的嵌入位置，并进行自回归文本生成。

| 组件 | 规格 |
|---|---|
| 文本骨干 | Qwen3-0.6B 风格因果解码器 |
| 音频编码器 | Whisper-Medium 编码器配置 |
| 音频前端 | `WhisperFeatureExtractor`，16 kHz，80 维 mel，30 秒切块 |
| 音频-文本桥接 | 4 倍时间维合并 + MLP 适配器 |
| 融合方式 | 音频特征替换 `<|audio_pad|>` 位置的嵌入 |
| 输出格式 | 紧凑式 `[start][Sxx]text[end]` 转录结果 |

## 输出格式

默认提示词要求模型按照如下格式输出每个语音片段：

```text
[start_time][Sxx]transcribed speech[end_time]
```

示例：

```text
[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
```

其中：

- `start_time` 和 `end_time` 是以秒为单位的时间戳。
- `[S01]`、`[S02]` 等是模型生成的匿名说话人标签。
- 说话人标签只表示输入音频内部的相对说话人，不代表真实身份。

## 适用场景

该模型适用于：

- 长音频自动语音识别。
- 说话人感知转写。
- 使用匿名标签进行说话人分离，例如 `[S01]`、`[S02]`、`[S03]`。
- 为会议、播客、访谈、课程、电话和视频生成带时间戳的转录结果。
- 需要同时获得文本内容和“谁在何时说话”信息的下游音频理解流程。

在医疗、法律、金融、安全关键或其他高风险场景中，不应在没有人工复核的情况下将该模型作为唯一事实来源。

## 快速开始

### 环境安装

本仓库使用自定义 Transformers 模型和处理器代码。加载模型和处理器时请始终设置 `trust_remote_code=True`。

代码要求 Python 3.10 或更高版本，并已在 Transformers 5.x 上测试。

```bash
python -m pip install -U pip
python -m pip install "transformers>=5.0.0,<6.0.0" "torch>=2.8" "torchaudio>=2.8"
python -m pip install "safetensors>=0.6.2" "numpy>=1.26,<2" "av>=14.0" "librosa>=0.11.0" "soundfile>=0.12" "soxr>=0.5" packaging
```

如需使用命令行推理和辅助工具，请克隆仓库并以 editable 模式安装：

```bash
git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
cd MOSS-Transcribe-Diarize
python -m pip install -e .
```

### 命令行推理

使用 greedy decoding：

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding greedy \
  --max-new-tokens 2048
```

使用采样解码：

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding sample \
  --temperature 0.7 \
  --max-new-tokens 2048
```

输出 JSON：

```bash
python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --json
```

音频文件会通过 Transformers 的音频加载器读取。MP4、MOV、MKV 等视频容器会通过 PyAV 解码，并在特征提取前重采样为单声道 16 kHz。

### Python 用法

```python
import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "/path/to/audio_or_video.mp4"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=True,
)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result["text"])
```

可以通过自定义 prompt 调整转写指令：

```python
messages = build_transcription_messages(
    audio_path,
    prompt="Please transcribe the audio with timestamps and speaker labels.",
)
```

命令行中也可以通过 `--prompt` 参数传入自定义提示词。

## 评测结果

我们使用字符错误率（CER）、拼接后最小排列字符错误率（cpCER）和 Delta-cp 评测 MOSS-Transcribe-Diarize。所有指标均为越低越好。横线（`-`）表示结果暂不可用。

| 数据集 | 指标 | Doubao | ElevenLabs | GPT-4o | Gemini 2.5 Pro | Gemini 3 Pro | VIBEVOICE ASR | MOSS Transcribe Diarize |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
| **AISHELL-4** | CER ↓ | 18.18 | 19.58 | - | 42.70 | 22.75 | 21.40 | **14.19** |
| | cpCER ↓ | 27.86 | 37.95 | - | 53.42 | 27.43 | 24.99 | **14.98** |
| | Delta-cp ↓ | 9.68 | 18.36 | - | 10.72 | 4.68 | 3.59 | **0.79** |
| **Podcast** | CER ↓ | 7.93 | 8.50 | - | 7.38 | - | 27.94 | **4.46** |
| | cpCER ↓ | 10.54 | 11.34 | - | 10.23 | - | 48.30 | **6.97** |
| | Delta-cp ↓ | 2.61 | 2.85 | - | 2.85 | - | 20.36 | **2.50** |
| **Movies** | CER ↓ | 9.94 | 11.49 | 14.37 | 15.46 | 8.62 | 14.59 | **6.58** |
| | cpCER ↓ | 30.88 | 17.85 | 23.67 | 24.15 | 14.73 | 42.54 | **13.68** |
| | Delta-cp ↓ | 20.94 | **6.37** | 9.31 | 8.69 | 6.11 | 27.94 | 7.24 |
| **Alimeeting** | CER ↓ | 25.25 | 25.70 | - | 27.43 | 26.75 | 27.40 | **24.80** |
| | cpCER ↓ | 37.57 | 36.69 | - | 41.64 | 32.84 | 29.33 | **21.51** |
| | Delta-cp ↓ | 12.31 | 10.99 | - | 14.21 | 6.09 | 1.93 | **-0.33** |

## 伦理说明

在转写、分离、存储或分享包含他人声音的录音前，用户应确保已获得适当授权或同意。该模型只生成匿名说话人标签，不验证真实身份。请勿将模型用于未授权监听、监控、监视或在缺乏充分保障和人工监督的情况下做出会对个人产生实质影响的决策。

## 许可证

本项目基于 Apache License 2.0 发布。

## 引用

公开模型发布后将补充引用信息。