Spaces:
Running
Running
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1"> | |
| <title>Fermata Transcriber</title> | |
| <meta name="description" content="Audio to beat-aligned multi-instrument MIDI, on a CPU."> | |
| <style> | |
| :root { --bg: #f7f7f5; --paper: #ffffff; --ink: #18181b; --muted: #5b5b66; --rule: #e3e3e0; --accent: #3446d6; | |
| --accent-ink: #ffffff; --win: #eef0ff; } | |
| @media (prefers-color-scheme: dark) { | |
| :root { --bg: #111114; --paper: #1a1a1f; --ink: #ececf0; --muted: #a1a1ab; --rule: #2c2c33; --accent: #8c98ff; | |
| --accent-ink: #111114; --win: #23264a; } | |
| } | |
| * { box-sizing: border-box; } | |
| body { margin: 0; background: var(--bg); color: var(--ink); font: 16px/1.6 system-ui, -apple-system, "Segoe UI", sans-serif; } | |
| main { max-width: 880px; margin: 0 auto; padding: 48px 16px 64px; } | |
| h1 { font-size: clamp(2rem, 6vw, 3rem); line-height: 1.1; margin: 0 0 12px; letter-spacing: -0.02em; } | |
| h2 { font-size: 1.25rem; margin: 48px 0 12px; } | |
| .lede { font-size: 1.15rem; color: var(--muted); max-width: 640px; margin: 0 0 24px; } | |
| .cta { display: inline-block; background: var(--accent); color: var(--accent-ink); text-decoration: none; font-weight: 600; | |
| padding: 12px 22px; border-radius: 999px; } | |
| .cta-note { color: var(--muted); font-size: 0.9rem; margin: 10px 0 0; } | |
| ul.points { padding-left: 1.1em; margin: 0; } | |
| ul.points li { margin: 6px 0; } | |
| figure { margin: 24px 0 0; background: var(--paper); border: 1px solid var(--rule); border-radius: 14px; padding: 12px; } | |
| figure img { width: 100%; height: auto; display: block; border-radius: 8px; background: #fff; } | |
| figcaption { color: var(--muted); font-size: 0.85rem; margin-top: 8px; } | |
| .table-wrap { overflow-x: auto; background: var(--paper); border: 1px solid var(--rule); border-radius: 14px; margin-top: 12px; } | |
| table { border-collapse: collapse; width: 100%; font-variant-numeric: tabular-nums; font-size: 0.95rem; } | |
| th, td { padding: 9px 12px; text-align: right; border-bottom: 1px solid var(--rule); white-space: nowrap; } | |
| th:first-child, td:first-child { text-align: left; } | |
| thead th { color: var(--muted); font-weight: 600; font-size: 0.85rem; } | |
| tbody tr:last-child td { border-bottom: 0; } | |
| tr.ours td { font-weight: 600; } | |
| td.best { background: var(--win); } | |
| .note { color: var(--muted); font-size: 0.88rem; margin: 10px 0 0; } | |
| footer { color: var(--muted); font-size: 0.85rem; margin-top: 56px; border-top: 1px solid var(--rule); padding-top: 16px; } | |
| a { color: var(--accent); } | |
| </style> | |
| </head> | |
| <body> | |
| <main> | |
| <h1>Fermata Transcriber</h1> | |
| <p class="lede">Audio in, <b>beat-aligned multi-instrument MIDI</b> out, on a CPU. Every note is placed on the song's bars | |
| and beats with a General MIDI instrument per track and drums, so the result opens as a usable score in a DAW or | |
| notation program.</p> | |
| <a class="cta" href="https://fermata.me.uk/#trial" target="_blank" rel="noopener">Try it, no sign-up</a> | |
| <p class="cta-note">First 60 seconds of your upload 路 3 songs a day 路 uploads are deleted after transcription</p> | |
| <figure> | |
| <img src="example_roll.png" alt="Piano roll of a transcribed song: several instruments and drums over 60 seconds" width="1320" height="495"> | |
| <figcaption>First 60 seconds of a Creative Commons recording, transcribed on 2 CPU threads in 17 seconds.</figcaption> | |
| </figure> | |
| <h2>What you get</h2> | |
| <ul class="points"> | |
| <li><b>synced MIDI</b>: notes on the beat grid, tempo follows the recording, so it lines up with the audio in a DAW</li> | |
| <li><b>score MIDI</b>: notes on the beat grid at one tempo with the meter, for notation programs</li> | |
| <li>about 78M parameters: audio encoder (28M) and a hybrid Gated DeltaNet / attention decoder (50M) running on llama.cpp</li> | |
| <li>a 150-second song takes about 135 seconds on a 4-core Intel N150 mini PC (8-bit weights)</li> | |
| </ul> | |
| <h2>Benchmarks</h2> | |
| <p class="note" style="margin-top:0">Note-level F1 (%), mir_eval, 50 ms onset tolerance, mean over pieces. MuScriptor was run by us on the | |
| same audio and scored with the same code. Higher is better; shaded = best in column. Fermata rows are v9, the version behind the trial.</p> | |
| <div class="table-wrap"><table> | |
| <thead><tr><th>MulTTiPop test 路 real pop, 385 songs</th><th>Onset</th><th>Offset</th><th>Exact (instrument)</th><th>Multi-inst.</th><th>Drums</th></tr></thead> | |
| <tbody> | |
| <tr><td>MuScriptor small</td><td>35.8</td><td>17.8</td><td>15.2</td><td>17.3</td><td>30.3</td></tr> | |
| <tr><td>MuScriptor medium</td><td class="best">41.3</td><td class="best">20.5</td><td>16.6</td><td class="best">18.9</td><td class="best">32.1</td></tr> | |
| <tr class="ours"><td>Fermata Transcriber</td><td>40.5</td><td>18.5</td><td class="best">18.9</td><td>18.2</td><td>31.8</td></tr> | |
| </tbody> | |
| </table></div> | |
| <div class="table-wrap"><table> | |
| <thead><tr><th>PHENICX-Anechoic 路 orchestra, 4 pieces</th><th>Onset</th><th>Offset</th><th>Multi-inst.</th></tr></thead> | |
| <tbody> | |
| <tr><td>MuScriptor small</td><td>50.1</td><td>34.9</td><td>6.2</td></tr> | |
| <tr><td>MuScriptor medium</td><td>54.4</td><td class="best">39.7</td><td>0.0</td></tr> | |
| <tr class="ours"><td>Fermata Transcriber</td><td class="best">61.0</td><td>31.2</td><td class="best">22.8</td></tr> | |
| </tbody> | |
| </table></div> | |
| <div class="table-wrap"><table> | |
| <thead><tr><th>Slakh2100 test 路 synthesized, 50 songs</th><th>Onset</th><th>Offset</th><th>Exact</th><th>Multi-inst.</th><th>Drums</th></tr></thead> | |
| <tbody> | |
| <tr><td>MuScriptor small</td><td>50.9</td><td>23.8</td><td>17.7</td><td>21.1</td><td>39.7</td></tr> | |
| <tr><td>MuScriptor medium</td><td>54.0</td><td>26.0</td><td>18.9</td><td>22.3</td><td>41.9</td></tr> | |
| <tr class="ours"><td>Fermata Transcriber</td><td class="best">62.5</td><td class="best">31.6</td><td class="best">34.8</td><td class="best">32.6</td><td class="best">52.2</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p class="note">Slakh bass references were shifted down one octave to match the audio. Inference settings were chosen | |
| on these test sets (no held-out split), so treat gaps of about 1 point as noise.</p> | |
| <h2>Limitations</h2> | |
| <ul class="points"> | |
| <li>Instrument labels are the weakest part; pitch and rhythm are much more reliable than the GM program choice.</li> | |
| <li>Note ends trail note starts. Vocals are not written as a melody track.</li> | |
| <li>Bar lines follow the beat tracker, so rubato or meter changes can shift downbeats.</li> | |
| </ul> | |
| <footer> | |
| Non-commercial use (CC BY-NC 4.0). Beat tracking by <a href="https://github.com/CPJKU/beat_this" target="_blank" rel="noopener">beat_this</a> (CPJKU, MIT), | |
| decoding on <a href="https://github.com/ggml-org/llama.cpp" target="_blank" rel="noopener">llama.cpp</a> (MIT). Part of | |
| <a href="https://fermata.me.uk" target="_blank" rel="noopener">Fermata</a>. | |
| </footer> | |
| </main> | |
| </body> | |
| </html> | |