v2.0.0 — Transkun transformer-only ONNX export + decode spec (from audio-claudio)

b445981 verified 1 day ago

3.16 kB

license: mit
library_name: onnxruntime
tags:
  - audio
  - music
  - piano-transcription
  - amt
  - onnx
  - transkun
  - semi-crf
pipeline_tag: audio-to-audio

Transkun — transformer-only ONNX export + decode spec

A self-contained ONNX export of Transkun (Yujia Yan's Neural Semi-CRF piano transcriber, 0.984 MAESTRO note F1) that runs the full model in-process with no Python/PyTorch at runtime. This is not a drop-in .onnx transcriber: torch.fft.rfft and the custom semi-CRF backtracking are not ONNX-exportable, so the mel front end and the Viterbi decode are provided as a documented decode spec + a reference decoder.

Attribution. The model, weights and architecture are the work of Yujia Yan, Frank Cwitkowitz and Zhiyao Duan ("Skipping the Frame-Level: Event-Based Piano Transcription with Neural Semi-CRFs", NeurIPS 2021). This is an independent export + decode spec, not affiliated with or endorsed by the authors. Upstream: https://github.com/Yujia-Yan/Skipping-The-Frame-Level. License: MIT (© 2021 Yujia Yan).

What's in the package

File	Role
`transkun.onnx` (~53 MB, opset 17)	`featuresBatch [1,T,229,6] → (S [T,T,90], ctx [90,T,256])` — the transformer + semi-CRF scorer + backbone features
`transkun-heads.onnx` (~3.4 MB)	`attr [N,768] → (velLogits [N,128], ofRaw [N,4])` — velocity + sub-frame onset/offset heads
`freq2mels.f32 [2049,229]`, `windows.f32 [6,4096]`, `symbols.i32 [90]`, `params.json`	frozen front-end constants
`LICENSE.transkun`	upstream MIT license
`export_transkun.py`, `export_transkun_heads.py`	regeneration scripts (need the `transkun` PyTorch package)

The decode spec (DECODE_SPEC.md) documents the mel front end, the S layout, the 90-track symbol map ([-64, -67, 21..108] = sustain/soft pedal + MIDI 21–108), the semi-CRF viterbiBackward, the 16 s/8 s segment stitching, and the attribute heads (velocity = argmax; ofValue = ContinuousBernoulli mean).

Reference decoder + validation

The reference decoder is the C# implementation in audio-claudio (mel front end, SemiCrfViterbi, TranskunTranscriber). It is validated note-identical to the native transkun CLI (PyTorch): on the test clips it reaches 100% note-level F1 at ±25 ms with exact velocity on every note — the export + decode spec reproduce the reference implementation, not merely approximate it.

Pipeline (how to run)

audio (mono, 44.1 kHz)
  → mel front end (framing 4096/1024, 6 windows, rfft ortho, freq2mels, log-norm) → featuresBatch
  → transkun.onnx → (S, ctx)
  → semi-CRF viterbiBackward(S) → per-track note intervals, over 16 s/8 s stitched segments
  → gather ctx at interval endpoints → transkun-heads.onnx → velocity + sub-frame onset/offset
  → notes (+ sustain/soft pedal from tracks 0/1)

See the repo's decode spec and TranskunTranscriber for the exact arithmetic (segment padding, forcedStartPos carry, merge, resolveOverlapping).