README.md

---
base_model: Qwen/Qwen3-ASR-1.7B
language:
- zh
license: apache-2.0
pipeline_tag: automatic-speech-recognition
tags:
- audio
- music
- singing-voice-transcription
- automatic-singing-transcription
- qwen3-asr
- asr
---

# VocalParse-1.7B

VocalParse is a unified singing voice transcription (SVT) model built upon a Large Audio Language Model (LALM). Fine-tuned from [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), it transcribes singing audio into a structured autoregressive token sequence that jointly encodes lyrics, pitch, note values, and global tempo (BPM).

- **Paper:** [VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models](https://huggingface.co/papers/2605.04613)
- **Repository:** [github.com/pymaster17/VocalParse](https://github.com/pymaster17/VocalParse)

```text
Singing Audio (16kHz) → Whisper Encoder → Qwen LLM Decoder → AST Token Sequence

感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>
```

## Usage

### Installation

It is recommended to use [uv](https://docs.astral.sh/uv/) for setup:

```bash
uv venv --python 3.10
source .venv/bin/activate
uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
uv pip install git+https://github.com/pymaster17/VocalParse.git
```

### Quick Inference

```python
from vocalparse import transcribe_one

text = transcribe_one(
    audio="path/to/song.wav",
    checkpoint="pymaster/VocalParse",
)
print(text)
# Example output: 感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> ... <BPM_89>
```

## Model Details

| Property | Value |
|---|---|
| **Base model** | Qwen3-ASR-1.7B (Whisper encoder + Qwen LLM decoder) |
| **Fine-tuning task** | Automatic Singing Transcription (AST) |
| **Training mode** | CoT (`asr_cot=true`, `bpm_position=last`) |
| **New vocabulary tokens** | ~400 AST tokens (pitch, note value, BPM) |
| **Input** | Mono 16 kHz singing audio |
| **Output** | Interleaved lyric + pitch + note sequence with global BPM |

### AST Token Vocabulary Extension

The base Qwen3-ASR vocabulary is extended with:
- **Pitch:** 128 tokens (`<P_0>` – `<P_127>`) representing MIDI notes.
- **Note value:** 12 tokens (e.g., `<NOTE_4>`, `<NOTE_8>`, `<NOTE_DOT_8>`).
- **Tempo:** 256 tokens (`<BPM_0>` – `<BPM_255>`).

### Output Format

- **Standard interleaved format** (`bpm_position=last`):
  `感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>`
  
- **CoT format** produced during generation (`asr_cot=true`): the model first outputs plain lyrics, then the full interleaved score, separated by `<|file_sep|>`:
  `感受到<|file_sep|>感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>`

## Evaluation Metrics

Metrics are computed with two-stage Needleman-Wunsch alignment: word-level alignment for lyrics, then pair-level alignment inside each matched word for pitch and note.

- **CER:** Character error rate on lyrics (silence tokens excluded).
- **Pitch MAE:** Mean absolute pitch error in MIDI semitones.
- **Note MAE:** Mean absolute error in log₂ note-value space.
- **BPM MAE:** Mean absolute tempo error.

## Limitations

- Primarily trained on Mandarin Chinese singing.
- Physical note durations are not predicted by this checkpoint.
- Long audio segments (> 30s) should be pre-segmented before inference.

## Citation

```bibtex
@article{vocalparse2026,
  title   = {VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models},
  author  = {Yukun Chen and Tianrui Wang and Zhaoxi Mu and Xinyu Yang and EngSiong Chng},
  journal = {arXiv preprint arXiv:2605.04613},
  year    = {2026},
  url     = {http://arxiv.org/abs/2605.04613}
}
```

## License

This model is licensed under Apache 2.0.