Audio-Text-to-Text
Transformers
Safetensors
English
Chinese
moss_transcribe_diarize
text2text-generation
moss
audio
speech
asr
diarization
timestamp-asr
long-form-audio
multimodal
multilingual
custom_code
Eval Results
Instructions to use OpenMOSS-Team/MOSS-Transcribe-Diarize with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use OpenMOSS-Team/MOSS-Transcribe-Diarize with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("OpenMOSS-Team/MOSS-Transcribe-Diarize", trust_remote_code=True) model = AutoModelForMultimodalLM.from_pretrained("OpenMOSS-Team/MOSS-Transcribe-Diarize", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README model capabilities
Browse files
README.md
CHANGED
|
@@ -28,7 +28,7 @@ pipeline_tag: audio-text-to-text
|
|
| 28 |
|
| 29 |
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness.
|
| 30 |
|
| 31 |
-
It supports transcription and diarization across 50+ languages.
|
| 32 |
|
| 33 |
Given an audio or video file, the model generates a compact speaker-aware transcript in one pass, including timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and beyond.
|
| 34 |
|
|
@@ -44,7 +44,6 @@ Given an audio or video file, the model generates a compact speaker-aware transc
|
|
| 44 |
- [Quickstart](#quickstart)
|
| 45 |
- [Environment Setup](#environment-setup)
|
| 46 |
- [Python Usage](#python-usage)
|
| 47 |
-
- [Custom Prompt and Hotwords](#custom-prompt-and-hotwords)
|
| 48 |
- [Serve with vLLM and SGLang](#serve-with-vllm-and-sglang)
|
| 49 |
- [Subtitle Web App](#subtitle-web-app)
|
| 50 |
- [Output Format](#output-format)
|
|
@@ -70,14 +69,6 @@ Core capabilities:
|
|
| 70 |
<img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize 0.9B model architecture" width="900">
|
| 71 |
</p>
|
| 72 |
|
| 73 |
-
| Component | Specification |
|
| 74 |
-
|---|---|
|
| 75 |
-
| Text backbone | Qwen3-0.6B style causal decoder |
|
| 76 |
-
| Audio encoder | Whisper-Medium encoder configuration |
|
| 77 |
-
| Audio frontend | `WhisperFeatureExtractor`, 16 kHz, 80 mel bins, 30 s chunks |
|
| 78 |
-
| Audio-text bridge | 4x temporal merge + MLP adaptor |
|
| 79 |
-
| Fusion | Audio features replace <code><|audio_pad|></code> embeddings via `masked_scatter` |
|
| 80 |
-
| Output format | Compact `[start][Sxx]text[end]` transcript with speaker tags such as `[S01]` |
|
| 81 |
|
| 82 |
This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`.
|
| 83 |
|
|
@@ -308,27 +299,6 @@ curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
|
| 308 |
-F response_format=verbose_json
|
| 309 |
```
|
| 310 |
|
| 311 |
-
```python
|
| 312 |
-
import requests
|
| 313 |
-
|
| 314 |
-
with open("audio.wav", "rb") as f:
|
| 315 |
-
resp = requests.post(
|
| 316 |
-
"http://localhost:8000/v1/audio/transcriptions",
|
| 317 |
-
data={
|
| 318 |
-
"model": "OpenMOSS-Team/MOSS-Transcribe-Diarize",
|
| 319 |
-
"response_format": "verbose_json",
|
| 320 |
-
},
|
| 321 |
-
files={"file": ("audio.wav", f, "audio/wav")},
|
| 322 |
-
timeout=300,
|
| 323 |
-
)
|
| 324 |
-
|
| 325 |
-
resp.raise_for_status()
|
| 326 |
-
payload = resp.json()
|
| 327 |
-
print(payload["text"])
|
| 328 |
-
for segment in payload.get("segments", []):
|
| 329 |
-
print(f"[{segment['start']:.2f}-{segment['end']:.2f}] {segment['text']}")
|
| 330 |
-
```
|
| 331 |
-
|
| 332 |
For longer multi-speaker audio, raise `max_new_tokens` so the decoder can finish the full diarized transcript:
|
| 333 |
|
| 334 |
```bash
|
|
@@ -339,37 +309,7 @@ curl -X POST http://localhost:8000/v1/audio/transcriptions \
|
|
| 339 |
-F max_new_tokens=65536
|
| 340 |
```
|
| 341 |
|
| 342 |
-
|
| 343 |
-
|---|---|---|---|
|
| 344 |
-
| `file` | file | required | Audio file uploaded as multipart form data |
|
| 345 |
-
| `model` | string | server default | Model identifier |
|
| 346 |
-
| `language` | string | unset | Optional language hint |
|
| 347 |
-
| `response_format` | string | `json` | `json`, `verbose_json`, or `text` |
|
| 348 |
-
| `temperature` | float | model default (`0.0`) | Sampling temperature |
|
| 349 |
-
| `max_new_tokens` | int | `5120` | Max generated tokens; raise for long audio, for example `65536` |
|
| 350 |
-
| `prompt` | string | unset | Optional instruction override; omit to use the built-in transcribe+diarize prompt |
|
| 351 |
-
|
| 352 |
-
For benchmarking, performance numbers, and implementation details, see the [SGLang Omni cookbook](https://github.com/sgl-project/sglang-omni/blob/main/docs/cookbook/moss_transcribe_diarize.md). The following single-H100 results are reported for short- and long-sequence multi-speaker ASR tasks.
|
| 353 |
-
|
| 354 |
-
`movies` short-sequence ASR:
|
| 355 |
-
|
| 356 |
-
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 357 |
-
|---:|---:|---:|---:|---:|
|
| 358 |
-
| 1 | 2.57 | 0.388 | 0.0612 | 29.76 |
|
| 359 |
-
| 2 | 4.89 | 0.409 | 0.0659 | 56.55 |
|
| 360 |
-
| 4 | 6.62 | 0.513 | 0.0790 | 76.64 |
|
| 361 |
-
| 8 | 6.80 | 0.533 | 0.0810 | 78.70 |
|
| 362 |
-
| 16 | 7.08 | 0.659 | 0.0922 | 81.98 |
|
| 363 |
-
|
| 364 |
-
`aishell4_long` long-sequence ASR:
|
| 365 |
-
|
| 366 |
-
| Concurrency | Throughput (req/s) | Mean latency (s) | RTF mean | audio_s/s |
|
| 367 |
-
|---:|---:|---:|---:|---:|
|
| 368 |
-
| 1 | 0.022 | 45.2 | 0.0197 | 50.64 |
|
| 369 |
-
| 2 | 0.032 | 60.7 | 0.0265 | 74.25 |
|
| 370 |
-
| 4 | 0.036 | 105.6 | 0.0461 | 81.64 |
|
| 371 |
-
| 8 | 0.040 | 172.6 | 0.0754 | 90.62 |
|
| 372 |
-
| 16 | 0.043 | 282.8 | 0.1237 | 98.83 |
|
| 373 |
|
| 374 |
### Subtitle Web App
|
| 375 |
|
|
|
|
| 28 |
|
| 29 |
MOSS-Transcribe-Diarize 0.9B is an end-to-end audio understanding model for long-form multi-speaker transcription, diarization, timestamps, and acoustic event awareness.
|
| 30 |
|
| 31 |
+
It supports transcription and diarization across 50+ languages, single-pass inference on audio recordings up to 90 minutes long, and custom hotword prompting for domain-specific terms.
|
| 32 |
|
| 33 |
Given an audio or video file, the model generates a compact speaker-aware transcript in one pass, including timestamps and anonymous speaker labels such as `[S01]`, `[S02]`, and beyond.
|
| 34 |
|
|
|
|
| 44 |
- [Quickstart](#quickstart)
|
| 45 |
- [Environment Setup](#environment-setup)
|
| 46 |
- [Python Usage](#python-usage)
|
|
|
|
| 47 |
- [Serve with vLLM and SGLang](#serve-with-vllm-and-sglang)
|
| 48 |
- [Subtitle Web App](#subtitle-web-app)
|
| 49 |
- [Output Format](#output-format)
|
|
|
|
| 69 |
<img src="Model_Architecture.png" alt="MOSS-Transcribe-Diarize 0.9B model architecture" width="900">
|
| 70 |
</p>
|
| 71 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 72 |
|
| 73 |
This Hugging Face repository includes the custom Transformers remote code required to load the model with `trust_remote_code=True`.
|
| 74 |
|
|
|
|
| 299 |
-F response_format=verbose_json
|
| 300 |
```
|
| 301 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 302 |
For longer multi-speaker audio, raise `max_new_tokens` so the decoder can finish the full diarized transcript:
|
| 303 |
|
| 304 |
```bash
|
|
|
|
| 309 |
-F max_new_tokens=65536
|
| 310 |
```
|
| 311 |
|
| 312 |
+
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 313 |
|
| 314 |
### Subtitle Web App
|
| 315 |
|