Hugging Face's logo

OpenMOSS-Team
/
MOSS-Transcribe-Diarize

Audio-Text-to-Text
Transformers
Safetensors
English
Chinese
moss_transcribe_diarize
text-generation
moss
audio
speech
asr
diarization
timestamp-asr
long-form-audio
multimodal
multilingual
custom_code
Model card Files
xet
 Community
15
MOSS-Transcribe-Diarize / README.md
zylin12's picture
zylin12
Fix README contents links
af440c1 verified
preview code
|
Raw
History Blame
8.89 kB
metadata 
license: apache-2.0
language:
  - en
  - zh
tags:
  - moss
  - audio
  - speech
  - asr
  - diarization
  - timestamp-asr
  - long-form-audio
  - multimodal
pipeline_tag: audio-text-to-text

MOSS-Transcribe-Diarize

MOSS-Transcribe-Diarize is an open-source speech transcription and diarization model from the OpenMOSS team. It performs unified modeling over long-form, multi-speaker audio, supporting automatic speech recognition, speaker-aware transcription, speaker diarization, timestamp prediction, and compact transcript generation.

Given an audio or video file, MOSS-Transcribe-Diarize generates a speaker-aware transcript in one pass, with segment timestamps and anonymous speaker labels such as [S01], [S02], and [S03].

News

  • 2026.05.21: We have released MOSS-Transcribe-Diarize.

Contents

Introduction

Long-form speech understanding requires more than plain transcription. For meetings, calls, podcasts, interviews, lectures, videos, and other real-world recordings, a useful transcript should identify what was said, who said it, and when it was said.

MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model.

  • Long-form ASR: Transcribes long audio and video recordings into text.
  • Speaker-aware transcription: Adds anonymous speaker labels to each speech segment.
  • Speaker diarization: Produces "who spoke when" style output without a separate diarization pipeline.
  • Timestamp prediction: Generates segment-level start and end timestamps.
  • Audio and video input: Supports common audio files and video containers decoded through PyAV.
  • Promptable generation: Allows users to customize the transcription instruction.

Model Architecture

MOSS-Transcribe-Diarize follows a modular audio-language design comprising three components: an audio encoder, a modality adapter, and a causal language model. Raw audio is converted into log-mel features, encoded by a Whisper-style audio encoder, projected into the language model embedding space through an MLP adapter, and then consumed by a Qwen3-style causal decoder for auto-regressive text generation.

The model uses audio placeholder tokens in the text sequence. During the forward pass, projected audio representations replace the corresponding placeholder embeddings, allowing the language model to generate timestamped, speaker-aware transcripts conditioned on the input audio.

Component Specification
Text backbone Qwen3-0.6B style causal decoder
Audio encoder Whisper-Medium encoder configuration
Audio frontend WhisperFeatureExtractor, 16 kHz, 80 mel bins, 30 s chunks
Audio-text adapter 4x temporal merge + MLP adapter
Fusion method Audio features replace `<
Output format Compact [start][Sxx]text[end] transcript

Released Models

Model Audio Encoder LLM Backbone Hugging Face
MOSS-Transcribe-Diarize Whisper-style audio encoder Qwen3-0.6B style decoder Hugging Face

More model variants may be released in the future. Stay tuned!

Evaluation

We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (-) indicates unavailable results.

Dataset Metric Doubao ElevenLabs GPT-4o Gemini 2.5 Pro Gemini 3 Pro VIBEVOICE ASR MOSS Transcribe Diarize
AISHELL-4 CER down 18.18 19.58 - 42.70 22.75 21.40 14.19
cpCER down 27.86 37.95 - 53.42 27.43 24.99 14.98
Delta-cp down 9.68 18.36 - 10.72 4.68 3.59 0.79
Podcast CER down 7.93 8.50 - 7.38 - 27.94 4.46
cpCER down 10.54 11.34 - 10.23 - 48.30 6.97
Delta-cp down 2.61 2.85 - 2.85 - 20.36 2.50
Movies CER down 9.94 11.49 14.37 15.46 8.62 14.59 6.58
cpCER down 30.88 17.85 23.67 24.15 14.73 42.54 13.68
Delta-cp down 20.94 6.37 9.31 8.69 6.11 27.94 7.24
Alimeeting CER down 25.25 25.70 - 27.43 26.75 27.40 24.80
cpCER down 37.57 36.69 - 41.64 32.84 29.33 21.51
Delta-cp down 12.31 10.99 - 14.21 6.09 1.93 -0.33

Quickstart

Environment Setup

We recommend Python 3.12 with a clean Conda environment. This repository uses custom Transformers model and processor code, so always load the model and processor with trust_remote_code=True.

git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
cd MOSS-Transcribe-Diarize

conda create -n moss-transcribe-diarize python=3.12 -y
conda activate moss-transcribe-diarize

conda install -c conda-forge "ffmpeg=7" -y
pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"

Optional: if your GPU supports FlashAttention 2, install the optional runtime with:

pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"

Command Line Inference

Run greedy decoding:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding greedy \
  --max-new-tokens 2048

Run sampling decoding:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --decoding sample \
  --temperature 0.7 \
  --max-new-tokens 2048

Return JSON output:

python infer.py \
  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
  --audio /path/to/audio_or_video.mp4 \
  --json

Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.

Python Usage 

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

from moss_transcribe_diarize.inference_utils import (
    build_transcription_messages,
    generate_transcription,
    resolve_device,
)

model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
audio_path = "/path/to/audio_or_video.mp4"

device = resolve_device("auto")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype="auto",
).to(dtype=dtype).to(device).eval()

processor = AutoProcessor.from_pretrained(
    model_id,
    trust_remote_code=True,
    fix_mistral_regex=True,
)

messages = build_transcription_messages(audio_path)
result = generate_transcription(
    model,
    processor,
    messages,
    max_new_tokens=2048,
    do_sample=False,
    device=device,
    dtype=dtype,
)

print(result["text"])

The default prompt asks the model to output each speech segment with a start timestamp, a speaker label, transcript text, and an end timestamp. You can customize the instruction with:

messages = build_transcription_messages(
    audio_path,
    prompt="Please transcribe the audio with timestamps and speaker labels.",
)

The same can be done from the command line with --prompt.

Output Format

The canonical output format is:

[start_time][Sxx]transcribed speech[end_time]

Example:

[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]

In this format:

  • start_time and end_time are timestamps in seconds.
  • [S01], [S02], and similar labels are anonymous model-generated speaker labels.
  • Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.

More Information

LICENSE

MOSS-Transcribe-Diarize is licensed under the Apache License 2.0.

Citation 

@misc{mosstranscribediarize2026,
      title={MOSS-Transcribe-Diarize},
      author={OpenMOSS Team},
      year={2026},
      howpublished={\url{https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize}},
      note={Hugging Face model repository}
}