| --- |
| license: mit |
| library_name: onnxruntime |
| tags: |
| - audio |
| - music |
| - piano-transcription |
| - amt |
| - onnx |
| - transkun |
| - semi-crf |
| pipeline_tag: audio-to-audio |
| --- |
| |
| # Transkun — transformer-only ONNX export + decode spec |
|
|
| A **self-contained ONNX export of [Transkun](https://github.com/Yujia-Yan/Skipping-The-Frame-Level)** |
| (Yujia Yan's Neural Semi-CRF piano transcriber, 0.984 MAESTRO note F1) that runs the full model **in-process |
| with no Python/PyTorch at runtime**. This is **not** a drop-in `.onnx` transcriber: `torch.fft.rfft` and the |
| custom semi-CRF backtracking are not ONNX-exportable, so the mel front end and the Viterbi decode are provided |
| as a **documented decode spec** + a **reference decoder**. |
|
|
| > **Attribution.** The model, weights and architecture are the work of **Yujia Yan, Frank Cwitkowitz and |
| > Zhiyao Duan** (*"Skipping the Frame-Level: Event-Based Piano Transcription with Neural Semi-CRFs"*, NeurIPS |
| > 2021). This is an independent export + decode spec, not affiliated with or endorsed by the authors. Upstream: |
| > <https://github.com/Yujia-Yan/Skipping-The-Frame-Level>. License: **MIT** (© 2021 Yujia Yan). |
|
|
| ## What's in the package |
|
|
| | File | Role | |
| |---|---| |
| | `transkun.onnx` (~53 MB, opset 17) | `featuresBatch [1,T,229,6] → (S [T,T,90], ctx [90,T,256])` — the transformer + semi-CRF scorer + backbone features | |
| | `transkun-heads.onnx` (~3.4 MB) | `attr [N,768] → (velLogits [N,128], ofRaw [N,4])` — velocity + sub-frame onset/offset heads | |
| | `freq2mels.f32 [2049,229]`, `windows.f32 [6,4096]`, `symbols.i32 [90]`, `params.json` | frozen front-end constants | |
| | `LICENSE.transkun` | upstream MIT license | |
| | `export_transkun.py`, `export_transkun_heads.py` | regeneration scripts (need the `transkun` PyTorch package) | |
|
|
| The **decode spec** (`DECODE_SPEC.md`) documents the mel front end, the `S` layout, the 90-track |
| symbol map (`[-64, -67, 21..108]` = sustain/soft pedal + MIDI 21–108), the semi-CRF `viterbiBackward`, the |
| 16 s/8 s segment stitching, and the attribute heads (velocity = argmax; `ofValue` = ContinuousBernoulli mean). |
|
|
| ## Reference decoder + validation |
|
|
| The reference decoder is the C# implementation in **[audio-claudio](https://github.com/TuesdayCrowd/audio-claudio)** |
| (mel front end, `SemiCrfViterbi`, `TranskunTranscriber`). It is validated **note-identical to the native |
| `transkun` CLI (PyTorch)**: on the test clips it reaches **100% note-level F1 at ±25 ms** with **exact |
| velocity** on every note — the export + decode spec reproduce the reference implementation, not merely |
| approximate it. |
|
|
| ## Pipeline (how to run) |
|
|
| ``` |
| audio (mono, 44.1 kHz) |
| → mel front end (framing 4096/1024, 6 windows, rfft ortho, freq2mels, log-norm) → featuresBatch |
| → transkun.onnx → (S, ctx) |
| → semi-CRF viterbiBackward(S) → per-track note intervals, over 16 s/8 s stitched segments |
| → gather ctx at interval endpoints → transkun-heads.onnx → velocity + sub-frame onset/offset |
| → notes (+ sustain/soft pedal from tracks 0/1) |
| ``` |
|
|
| See the repo's decode spec and `TranskunTranscriber` for the exact arithmetic (segment padding, `forcedStartPos` |
| carry, merge, `resolveOverlapping`). |
|
|
