Add model architecture figure

README.md

@@ -0,0 +1,253 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+tags:
+- moss
+- audio
+- speech
+- asr
+- diarization
+- timestamp-asr
+- long-form-audio
+- multimodal
+pipeline_tag: audio-text-to-text
+---
+
+# MOSS-Transcribe-Diarize
+
+MOSS-Transcribe-Diarize is an open-source **speech transcription and diarization model** from the OpenMOSS team. It performs unified modeling over long-form, multi-speaker audio, supporting **automatic speech recognition, speaker-aware transcription, speaker diarization, timestamp prediction, and compact transcript generation**.
+
+Given an audio or video file, MOSS-Transcribe-Diarize generates a speaker-aware transcript in one pass, with segment timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and `[S03]`.
+
+## News
+
+* 2026.05.21: We have released MOSS-Transcribe-Diarize.
+
+## Contents
+
+- [Introduction](#introduction)
+- [Model Architecture](#model-architecture)
+- [Released Models](#released-models)
+- [Evaluation](#evaluation)
+- [Quickstart](#quickstart)
+  - [Environment Setup](#environment-setup)
+  - [Command Line Inference](#command-line-inference)
+  - [Python Usage](#python-usage)
+- [Output Format](#output-format)
+- [More Information](#more-information)
+- [LICENSE](#license)
+- [Citation](#citation)
+
+## Introduction
+
+Long-form speech understanding requires more than plain transcription. For meetings, calls, podcasts, interviews, lectures, videos, and other real-world recordings, a useful transcript should identify **what was said**, **who said it**, and **when it was said**.
+
+MOSS-Transcribe-Diarize is built to unify these capabilities within a single generative model.
+
+* **Long-form ASR**: Transcribes long audio and video recordings into text.
+* **Speaker-aware transcription**: Adds anonymous speaker labels to each speech segment.
+* **Speaker diarization**: Produces "who spoke when" style output without a separate diarization pipeline.
+* **Timestamp prediction**: Generates segment-level start and end timestamps.
+* **Audio and video input**: Supports common audio files and video containers decoded through PyAV.
+* **Promptable generation**: Allows users to customize the transcription instruction.
+
+## Model Architecture
+
+<p align="center">
+  <img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize model architecture" width="900">
+</p>
+
+MOSS-Transcribe-Diarize follows a modular audio-language design comprising three components: an audio encoder, a modality adapter, and a causal language model. Raw audio is converted into log-mel features, encoded by a Whisper-style audio encoder, projected into the language model embedding space through an MLP adapter, and then consumed by a Qwen3-style causal decoder for auto-regressive text generation.
+
+The model uses audio placeholder tokens in the text sequence. During the forward pass, projected audio representations replace the corresponding placeholder embeddings, allowing the language model to generate timestamped, speaker-aware transcripts conditioned on the input audio.
+
+| Component | Specification |
+| --- | --- |
+| Text backbone | Qwen3-0.6B style causal decoder |
+| Audio encoder | Whisper-Medium encoder configuration |
+| Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
+| Audio-text adapter | 4x temporal merge + MLP adapter |
+| Fusion method | Audio features replace `<|audio_pad|>` embeddings |
+| Output format | Compact `[start][Sxx]text[end]` transcript |
+
+## Released Models
+
+| Model | Audio Encoder | LLM Backbone | Hugging Face |
+| --- | --- | --- | --- |
+| **MOSS-Transcribe-Diarize** | Whisper-style audio encoder | Qwen3-0.6B style decoder | [Hugging Face](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize) |
+
+> More model variants may be released in the future. Stay tuned!
+
+## Evaluation
+
+We evaluate MOSS-Transcribe-Diarize with Character Error Rate (CER), concatenated minimum-permutation Character Error Rate (cpCER), and Delta-cp. Lower is better for all metrics. A dash (`-`) indicates unavailable results.
+
+| Dataset | Metric | Doubao | ElevenLabs | GPT-4o | Gemini 2.5 Pro | Gemini 3 Pro | VIBEVOICE ASR | MOSS Transcribe Diarize |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **AISHELL-4** | CER down | 18.18 | 19.58 | - | 42.70 | 22.75 | 21.40 | **14.19** |
+| | cpCER down | 27.86 | 37.95 | - | 53.42 | 27.43 | 24.99 | **14.98** |
+| | Delta-cp down | 9.68 | 18.36 | - | 10.72 | 4.68 | 3.59 | **0.79** |
+| **Podcast** | CER down | 7.93 | 8.50 | - | 7.38 | - | 27.94 | **4.46** |
+| | cpCER down | 10.54 | 11.34 | - | 10.23 | - | 48.30 | **6.97** |
+| | Delta-cp down | 2.61 | 2.85 | - | 2.85 | - | 20.36 | **2.50** |
+| **Movies** | CER down | 9.94 | 11.49 | 14.37 | 15.46 | 8.62 | 14.59 | **6.58** |
+| | cpCER down | 30.88 | 17.85 | 23.67 | 24.15 | 14.73 | 42.54 | **13.68** |
+| | Delta-cp down | 20.94 | **6.37** | 9.31 | 8.69 | 6.11 | 27.94 | 7.24 |
+| **Alimeeting** | CER down | 25.25 | 25.70 | - | 27.43 | 26.75 | 27.40 | **24.80** |
+| | cpCER down | 37.57 | 36.69 | - | 41.64 | 32.84 | 29.33 | **21.51** |
+| | Delta-cp down | 12.31 | 10.99 | - | 14.21 | 6.09 | 1.93 | **-0.33** |
+
+## Quickstart
+
+### Environment Setup
+
+We recommend Python 3.12 with a clean Conda environment. This repository uses custom Transformers model and processor code, so always load the model and processor with `trust_remote_code=True`.
+
+```bash
+git clone https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize
+cd MOSS-Transcribe-Diarize
+
+conda create -n moss-transcribe-diarize python=3.12 -y
+conda activate moss-transcribe-diarize
+
+conda install -c conda-forge "ffmpeg=7" -y
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime]"
+```
+
+Optional: if your GPU supports FlashAttention 2, install the optional runtime with:
+
+```bash
+pip install --extra-index-url https://download.pytorch.org/whl/cu128 -e ".[torch-runtime,flash-attn]"
+```
+
+### Command Line Inference
+
+Run greedy decoding:
+
+```bash
+python infer.py \
+  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --audio /path/to/audio_or_video.mp4 \
+  --decoding greedy \
+  --max-new-tokens 2048
+```
+
+Run sampling decoding:
+
+```bash
+python infer.py \
+  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --audio /path/to/audio_or_video.mp4 \
+  --decoding sample \
+  --temperature 0.7 \
+  --max-new-tokens 2048
+```
+
+Return JSON output:
+
+```bash
+python infer.py \
+  --model OpenMOSS-Team/MOSS-Transcribe-Diarize \
+  --audio /path/to/audio_or_video.mp4 \
+  --json
+```
+
+Audio files are loaded through the Transformers audio loader. Video containers such as MP4, MOV, and MKV are decoded with PyAV and resampled to mono 16 kHz before feature extraction.
+
+### Python Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+from moss_transcribe_diarize.inference_utils import (
+    build_transcription_messages,
+    generate_transcription,
+    resolve_device,
+)
+
+model_id = "OpenMOSS-Team/MOSS-Transcribe-Diarize"
+audio_path = "/path/to/audio_or_video.mp4"
+
+device = resolve_device("auto")
+dtype = torch.bfloat16 if device.type == "cuda" else torch.float32
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype="auto",
+).to(dtype=dtype).to(device).eval()
+
+processor = AutoProcessor.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    fix_mistral_regex=True,
+)
+
+messages = build_transcription_messages(audio_path)
+result = generate_transcription(
+    model,
+    processor,
+    messages,
+    max_new_tokens=2048,
+    do_sample=False,
+    device=device,
+    dtype=dtype,
+)
+
+print(result["text"])
+```
+
+The default prompt asks the model to output each speech segment with a start timestamp, a speaker label, transcript text, and an end timestamp. You can customize the instruction with:
+
+```python
+messages = build_transcription_messages(
+    audio_path,
+    prompt="Please transcribe the audio with timestamps and speaker labels.",
+)
+```
+
+The same can be done from the command line with `--prompt`.
+
+## Output Format
+
+The canonical output format is:
+
+```text
+[start_time][Sxx]transcribed speech[end_time]
+```
+
+Example:
+
+```text
+[0.48][S01]Welcome everyone[1.66][12.26][S02]The new transcription pipeline is ready for evaluation[13.81][14.36][S01]Great, include the diarization results in the report[18.76]
+```
+
+In this format:
+
+* `start_time` and `end_time` are timestamps in seconds.
+* `[S01]`, `[S02]`, and similar labels are anonymous model-generated speaker labels.
+* Speaker labels are relative labels within the input audio and should not be interpreted as real speaker identities.
+
+## More Information
+
+* **MOSI.AI**: <https://mosi.cn>
+* **OpenMOSS**: <https://www.open-moss.com>
+
+## LICENSE
+
+MOSS-Transcribe-Diarize is licensed under the Apache License 2.0.
+
+## Citation
+
+```bibtex
+@misc{mosstranscribediarize2026,
+      title={MOSS-Transcribe-Diarize},
+      author={OpenMOSS Team},
+      year={2026},
+      howpublished={\url{https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize}},
+      note={Hugging Face model repository}
+}
+```