File size: 3,775 Bytes
3fc1146 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 | ---
base_model: Qwen/Qwen3-ASR-1.7B
language:
- zh
license: apache-2.0
pipeline_tag: automatic-speech-recognition
tags:
- audio
- music
- singing-voice-transcription
- automatic-singing-transcription
- qwen3-asr
- asr
---
# VocalParse-1.7B
VocalParse is a unified singing voice transcription (SVT) model built upon a Large Audio Language Model (LALM). Fine-tuned from [Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), it transcribes singing audio into a structured autoregressive token sequence that jointly encodes lyrics, pitch, note values, and global tempo (BPM).
- **Paper:** [VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models](https://huggingface.co/papers/2605.04613)
- **Repository:** [github.com/pymaster17/VocalParse](https://github.com/pymaster17/VocalParse)
```text
Singing Audio (16kHz) → Whisper Encoder → Qwen LLM Decoder → AST Token Sequence
感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>
```
## Usage
### Installation
It is recommended to use [uv](https://docs.astral.sh/uv/) for setup:
```bash
uv venv --python 3.10
source .venv/bin/activate
uv pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu124
uv pip install git+https://github.com/pymaster17/VocalParse.git
```
### Quick Inference
```python
from vocalparse import transcribe_one
text = transcribe_one(
audio="path/to/song.wav",
checkpoint="pymaster/VocalParse",
)
print(text)
# Example output: 感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> ... <BPM_89>
```
## Model Details
| Property | Value |
|---|---|
| **Base model** | Qwen3-ASR-1.7B (Whisper encoder + Qwen LLM decoder) |
| **Fine-tuning task** | Automatic Singing Transcription (AST) |
| **Training mode** | CoT (`asr_cot=true`, `bpm_position=last`) |
| **New vocabulary tokens** | ~400 AST tokens (pitch, note value, BPM) |
| **Input** | Mono 16 kHz singing audio |
| **Output** | Interleaved lyric + pitch + note sequence with global BPM |
### AST Token Vocabulary Extension
The base Qwen3-ASR vocabulary is extended with:
- **Pitch:** 128 tokens (`<P_0>` – `<P_127>`) representing MIDI notes.
- **Note value:** 12 tokens (e.g., `<NOTE_4>`, `<NOTE_8>`, `<NOTE_DOT_8>`).
- **Tempo:** 256 tokens (`<BPM_0>` – `<BPM_255>`).
### Output Format
- **Standard interleaved format** (`bpm_position=last`):
`感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>`
- **CoT format** produced during generation (`asr_cot=true`): the model first outputs plain lyrics, then the full interleaved score, separated by `<|file_sep|>`:
`感受到<|file_sep|>感 <P_68> <NOTE_4> 受 <P_60> <NOTE_8> 到 <P_65> <NOTE_8> ... <BPM_89>`
## Evaluation Metrics
Metrics are computed with two-stage Needleman-Wunsch alignment: word-level alignment for lyrics, then pair-level alignment inside each matched word for pitch and note.
- **CER:** Character error rate on lyrics (silence tokens excluded).
- **Pitch MAE:** Mean absolute pitch error in MIDI semitones.
- **Note MAE:** Mean absolute error in log₂ note-value space.
- **BPM MAE:** Mean absolute tempo error.
## Limitations
- Primarily trained on Mandarin Chinese singing.
- Physical note durations are not predicted by this checkpoint.
- Long audio segments (> 30s) should be pre-segmented before inference.
## Citation
```bibtex
@article{vocalparse2026,
title = {VocalParse: Towards Unified and Scalable Singing Voice Transcription with Large Audio Language Models},
author = {Yukun Chen and Tianrui Wang and Zhaoxi Mu and Xinyu Yang and EngSiong Chng},
journal = {arXiv preprint arXiv:2605.04613},
year = {2026},
url = {http://arxiv.org/abs/2605.04613}
}
```
## License
This model is licensed under Apache 2.0. |