sheet-music-medium-d5

Whisper-style encoder-decoder for piano audio → restdur-tokenized notation. "Medium" size counterpart to sheet-music-small-d5; best overall run in the May 2026 sweep when judged on the mean across synthesized + real-piano evaluation clips (see project progress_log/5-15-2026.md).

Model summary

• Architecture: Whisper-style encoder-decoder, "medium" size
• Tokenizer: restdur — fused REST_DUR_X tokens for rests; pitch + duration tokens for notes. Vocab size 6244.
• Audio input: CQT, 216 bins, 10ms hop, 16 kHz mono.
• Training: CE loss only, AdamW peak LR 1.5e-4, dropout 0.2, grad clip 1.0, cosine schedule with 0.2-epoch warmup. SpecAugment on train split. 198 epochs on dataset 5 (noised synthesized piano).

Headline metrics (teacher-forced, real-audio clips)

Clip	note_acc	rest_kind_acc	note_dur_mae_10ms	rest_dur_mae_10ms
Alkaline	0.96	0.00	4.2	3000
VivaLaVida (synth)	0.95	0.56	9.5	1312
UsedToBeMine	0.33	0.00	15.1	0
VivaLaVidaReal (real singing)	0.44	0.50	19.5	1521

Best mean note_acc across the four real-audio clips in the sweep (0.67). Trades a little real-piano note_acc vs. the small_d5 baseline (0.44 vs. 0.56) for stronger note_acc on every synthetic real-music clip.

Files

• best.pt — model state_dict, ~134 MB
• eval_results.json — full eval metrics across all OOD + real-audio clips

Loading

import torch
# model.py + data.py from the sheet-music repo (matching commit) are
# required to instantiate WhisperLite with the restdur vocab.
import model
model.set_size('medium')
from model import WhisperLite
net = WhisperLite()
net.load_state_dict(torch.load('best.pt', map_location='cpu'), strict=False)
net.eval()