| --- |
| license: mit |
| library_name: pytorch |
| pipeline_tag: image-to-text |
| tags: [optical-music-recognition, omr, music, jianpu, sargam, image-to-text] |
| --- |
| |
| # omr-synth — a small OMR model for Western staff, Jianpu (简谱) & Sargam |
|
|
| A ~26M-parameter image→sequence Transformer (CNN encoder + autoregressive decoder) that reads a page of |
| music notation and writes out a compact symbolic score (OMRDSL). Trained **entirely on synthetic data, |
| overnight, on a Mac mini (M4, Apple-Silicon MPS)** — no cloud GPU. **A weekend learning project: early and |
| honest, not production OMR.** |
|
|
| - 💻 Code: https://github.com/abhid1234/omr-synth |
| - ▶️ Interactive demo: https://omr-synth.vercel.app |
| - 📚 Dataset: https://huggingface.co/datasets/abhid1234/omr-synth |
|
|
| ## What it does |
| Reads **Western staff, Jianpu, and Sargam** — three ways of writing music — into ONE shared symbolic target. |
| General vision models can't read Jianpu/Sargam and there's almost no labeled data for them; a render-forward |
| engine manufactures it (write the score first — that's the label — then render it into an image). |
|
|
| ## Honest held-out results (pages it never saw) |
| Scoped to **level-0** (clean, monophonic) to be learnable in one overnight run: |
|
|
| | Notation | Token accuracy | Content-only (the actual notes) | |
| |---|---|---| |
| | Western | ~30% | ~11% | |
| | Jianpu | ~28% | ~9% | |
| | Sargam | ~28% | ~8% | |
| | **Overall** | **~29%** | **~9%** | |
|
|
| *Token accuracy* counts every output token, including the easy structural scaffolding (clef, meter, |
| bar/voice layout) it almost always gets right. *Content-only* counts just the actual notes and durations — |
| the honest "can it read the music" number: about one note in ten. Exact-match ~0%. Balanced across all three. |
|
|
| ## Architecture |
| CNN stem (1→64→128→384) + 6-layer Transformer encoder (d=384) reading a grayscale, ink-positive, |
| aspect-padded page; 6-layer autoregressive Transformer decoder emitting OMRDSL. ~26M params |
| (`src/model/omr.py`). This checkpoint is the best of the overnight run (gentler-LR rerun, epoch 7). |
|
|
| ## Usage |
| ```bash |
| git clone https://github.com/abhid1234/omr-synth && cd omr-synth |
| pip install torch torchvision |
| python predict.py --checkpoint omr-synth-e7.pt --image page.png |
| ``` |
|
|
| ## Limitations |
| Level-0 only (no polyphony/dynamics/ornaments, no real manuscripts). Reads content partially. Training was |
| unstable early (fixed with a warmup+cosine schedule). The reusable contribution is the render-forward |
| **data engine**, not this early model. MIT · Abhi Das. |
|
|
