README.md

metadata

license: apache-2.0
language:
  - en
  - zh
library_name: transformers
pipeline_tag: audio-text-to-text
tags:
  - moss
  - moss_audio
  - moss_transcribe_diarize
  - audio
  - speech
  - asr
  - speaker-diarization
  - timestamp-asr
  - long-form-audio
  - multimodal
  - custom_code

MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize is an open-source speech transcription and diarization model from the OpenMOSS team. It performs unified modeling over long-form, multi-speaker audio, supporting automatic speech recognition, speaker-aware transcription, speaker diarization, timestamp prediction, and compact transcript generation.

Given an audio or video file, MOSS-Transcribe-Diarize generates a speaker-aware transcript in one pass, with segment timestamps and anonymous speaker labels such as [S01], [S02], and [S03].

News

• 2026.05.21: We have released MOSS-Transcribe-Diarize.

Contents

• Introduction
• Model Architecture
• Released Models
• Evaluation
• Quickstart
  • Environment Setup
  • Command Line Inference
  • Python Usage
• Output Format
• More Information
• LICENSE
• Citation

Introduction

Long-form speech understanding requires more than plain transcription. For meetings, calls, podcasts, interviews, lectures, videos, and other real-world recordings, a useful transcript should identify what was said, who said it, and when it was said.

MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model.

• Long-form ASR: Transcribes long audio and video recordings into text.
• Speaker-aware transcription: Adds anonymous speaker labels to each speech segment.
• Speaker diarization: Produces "who spoke when" style output without a separate diarization pipeline.
• Timestamp prediction: Generates segment-level start and end timestamps.
• Audio and video input: Supports common audio files and video containers decoded through PyAV.
• Promptable generation: Allows users to customize the transcription instruction.

Model Architecture

MOSS-Transcribe-Diarize follows a modular audio-language design comprising three components: an audio encoder, a modality adapter, and a causal language model. Raw audio is converted into log-mel features, encoded by a Whisper-style audio encoder, projected into the language model embedding space through an MLP adapter, and then consumed by a Qwen3-style causal decoder for auto-regressive text generation.

The model uses audio placeholder tokens in the text sequence. During the forward pass, projected audio representations replace the corresponding placeholder embeddings, allowing the language model to generate timestamped, speaker-aware transcripts conditioned on the input audio.

Component	Specification
Text backbone	Qwen3-0.6B style causal decoder
Audio encoder	Whisper-Medium encoder configuration
Audio frontend	WhisperFeatureExtractor, 16 kHz, 80 mel bins, 30 s chunks
Audio-text adapter	4x temporal merge + MLP adapter
Fusion method	Audio features replace `<
Output format	Compact [start][Sxx]text[end] transcript

Released Models

Model	Audio Encoder	LLM Backbone	Hugging Face
MOSS-Transcribe-Diarize	Whisper-style audio encoder	Qwen3-0.6B style decoder	Hugging Face

More model variants may be released in the future. Stay tuned!

Evaluation

We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (-) indicates unavailable results.

Dataset	Metric	Doubao	ElevenLabs	GPT-4o	Gemini 2.5 Pro	Gemini 3 Pro	VIBEVOICE ASR	MOSS Transcribe Diarize
AISHELL-4	CER down	18.18	19.58	-	42.70	22.75	21.40	14.19
	cpCER down	27.86	37.95	-	53.42	27.43	24.99	14.98
	Delta-cp down	9.68	18.36	-	10.72	4.68	3.59	0.79
Podcast	CER down	7.93	8.50	-	7.38	-	27.94	4.46
	cpCER down	10.54	11.34	-	10.23	-	48.30	6.97
	Delta-cp down	2.61	2.85	-	2.85	-	20.36	2.50
Movies	CER down	9.94	11.49	14.37	15.46	8.62	14.59	6.58
	cpCER down	30.88	17.85	23.67	24.15	14.73	42.54	13.68
	Delta-cp down	20.94	6.37	9.31	8.69	6.11	27.94	7.24
Alimeeting	CER down	25.25	25.70	-	27.43	26.75	27.40	24.80
	cpCER down	37.57	36.69	-	41.64	32.84	29.33	21.51
	Delta-cp down	12.31	10.99	-	14.21	6.09	1.93	-0.33

Quickstart

Environment Setup

We recommend Python 3.12 with a clean Conda environment. This repository uses custom Transformers model and processor code, so always load the model and processor with trust_remote_code=True.

git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
cd MOSS-Transcribe-Diarize

conda create -n moss-transcribe-diarize python=3.12 -y
conda activate moss-transcribe-diarize

conda install -c conda-forge "ffmpeg=7" -y
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"

Optional: if your GPU supports FlashAttention 2, install the optional runtime with:

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"

Command Line Inference

Run greedy decoding:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding greedy \
  --max-new-tokens 2048

Run sampling decoding:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding sample \
  --temperature 0.7 \
  --max-new-tokens 2048

Return JSON output:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --json

Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.

Python Usage

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "/path/to/audio_or_video.mp4"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=True,
)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result["text"])

The default prompt asks the model to output each speech segment with a start timestamp, a speaker label, transcript text, and an end timestamp. You can customize the instruction with:

messages = build_transcription_messages(
    audio_path,
    prompt="Please transcribe the audio with timestamps and speaker labels.",
)

The same can be done from the command line with --prompt.

Output Format

The canonical output format is:

[start_time][Sxx]transcribed speech[end_time]

Example:

[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]

In this format:

• start_time and end_time are timestamps in seconds.
• [S01], [S02], and similar labels are anonymous model-generated speaker labels.
• Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.

More Information

• MOSI.AI: https://mosi.cn
• OpenMOSS: https://www.open-moss.com

LICENSE

MOSS-Transcribe-Diarize is licensed under the Apache License 2.0.

Citation

@misc{mosstranscribediarize2026,
      title={MOSS-Transcribe-Diarize},
      author={OpenMOSS Team},
      year={2026},
      howpublished={\url{https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize}},
      note={Hugging Face model repository}
}