sheet-music-small-d5

Whisper-style encoder-decoder for piano audio → restdur-tokenized notation. Best run across the May 2026 sweep on real-piano evaluation (see eval_results.json and project progress_log/5-15-2026.md).

Model summary

• Architecture: Whisper-style encoder-decoder, "small" size (~15M params)
• Tokenizer: restdur — fused REST_DUR_X tokens for rests; pitch + duration tokens for notes. Vocab size 6244.
• Audio input: CQT, 216 bins, 10ms hop, 16 kHz mono.
• Training: CE loss only, AdamW peak LR 1.5e-4, dropout 0.2, grad clip 1.0, cosine schedule with 0.2-epoch warmup. SpecAugment on train split. 200 epochs on dataset 5 (noised synthesized piano).

Headline metrics (teacher-forced, real-audio clips)

Clip	note_acc	rest_kind_acc	note_dur_mae_10ms	rest_dur_mae_10ms
Alkaline	1.00	0.00	3.7	3000
VivaLaVida (synth)	0.83	0.22	89.8	2343
UsedToBeMine	0.24	0.00	21.3	0
VivaLaVidaReal (real)	0.56	0.50	17.1	1521

Best balanced model in the sweep — only one that handles both notes and rests on the real-piano clip without collapsing to all-rests or all-notes.

Files

• best.pt — model state_dict, ~58 MB
• eval_results.json — full eval metrics across all OOD + real-audio clips

Loading

import torch
# model.py + data.py from the sheet-music repo (matching commit) are
# required to instantiate WhisperLite with the restdur vocab.
import model
model.set_size('small')
from model import WhisperLite
net = WhisperLite()
net.load_state_dict(torch.load('best.pt', map_location='cpu'), strict=False)
net.eval()