| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - audio |
| - music |
| - multi-pitch-estimation |
| - music-information-retrieval |
| - causal |
| - arxiv:2610.33113 |
| --- |
| |
| # JIT-MPE and CRNN comparison checkpoints |
|
|
| Trained weights for the paper [JIT-MPE: Just-in-Time Multi-Pitch Estimation under Output Deadlines](https://arxiv.org/abs/2610.33113) |
| (Yoshiyuki Kobayashi, Frieve, 2026). |
| [Source code](https://github.com/Frieve-A/jit-mpe/tree/08bbd5acbb223962895f847a86c2d47a36179896) · |
| [Paper page](https://huggingface.co/papers/2610.33113) · [License](https://huggingface.co/Frieve-A/jit-mpe/blob/main/LICENSE) · [Data and software credits](THIRD_PARTY.md) |
|
|
| JIT-MPE is a small causal model (about 86k parameters) that estimates which of the 88 piano pitches are sounding in |
| mixed-ensemble audio. It processes the audio in 20 ms frames and can revise a frame until its output deadline |
| (for example, 50 ms). On one CPU thread it takes about 2.4 ms per frame. |
|
|
| This repository contains the same weights as the GitHub release, unchanged (no conversion or retraining). It also |
| includes the comparison CRNN models that the author trained for the paper. The external systems compared in the |
| paper are not included; get them from their own authors. |
|
|
| ## Models |
|
|
| | System ID | Model | Parameters | Type | |
| |---|---|---:|---| |
| | `S` | JIT-MPE | 86,425 | Causal, five delay heads | |
| | `A0` | Harmonic CRNN | 85,813 | Causal, one head | |
| | `A1d` | Dilated CRNN | 95,767 | Causal, one head | |
| | `B1` | Plain CRNN | 95,767 | Causal, one head | |
| | `A0-bi` | Harmonic BiCRNN | 83,729 | Bidirectional, full-length input only | |
| | `A1d-bi` | Dilated BiCRNN | 93,936 | Bidirectional, full-length input only | |
| | `B1-bi` | Plain BiCRNN | 93,936 | Bidirectional, full-length input only | |
|
|
| Each system was trained with seeds **1, 3 and 5**. These are three separate runs, not an ensemble; the paper reports |
| their mean. Parameter counts exclude the auxiliary onset head, which is used only in training (it is still stored |
| in the checkpoints). |
|
|
| ## Files |
|
|
| ```text |
| weights/<SYSTEM>-s<SEED>.safetensors # model weights (21 files) |
| weights/<SYSTEM>-s<SEED>.calibration.json # decision thresholds for each model (21 files) |
| weights/head-selection-s<SEED>.json # which delay head to use for each deadline (one per seed) |
| configs/<SYSTEM>.json # training settings (not Transformers configs) |
| requirements-paper.txt # Python environment used for the paper |
| release-manifest.json # size and SHA-256 of every file, source revision, paper metadata |
| LICENSE |
| THIRD_PARTY.md |
| CITATION.cff |
| ``` |
|
|
| To run one model you need three files: its `.safetensors`, its `.calibration.json` and the `head-selection` file for |
| the same seed. Keep them in the same folder. The model code is in the GitHub package, so `AutoModel` and the |
| Transformers `pipeline` cannot load these weights. |
|
|
| ## Usage |
|
|
| Requirements: Python 3.10 or later and PyTorch for your platform. Install the source code at the matching revision |
| and the Hugging Face Hub client: |
|
|
| ```bash |
| git clone https://github.com/Frieve-A/jit-mpe.git |
| cd jit-mpe |
| git checkout 08bbd5acbb223962895f847a86c2d47a36179896 |
| python -m pip install -e . |
| python -m pip install huggingface_hub==0.34.3 |
| ``` |
|
|
| Python example (downloads only the three files for `S`, seed 1): |
|
|
| ```python |
| from pathlib import Path |
| import torch |
| from huggingface_hub import snapshot_download |
| from jit_mpe import load |
| from jit_mpe.infer import estimate, read_audio, write |
| |
| name, seed = "S", 1 |
| snapshot = snapshot_download( |
| repo_id="Frieve-A/jit-mpe", |
| revision="main", # replace with a full Hub commit hash to pin a published snapshot |
| allow_patterns=[ |
| f"weights/{name}-s{seed}.safetensors", |
| f"weights/{name}-s{seed}.calibration.json", |
| f"weights/head-selection-s{seed}.json", |
| ], |
| ) |
| system = load(name, seed, weights_dir=Path(snapshot) / "weights", device="cpu") |
| torch.set_num_threads(1) |
| with torch.inference_mode(): |
| times, frequencies = estimate( |
| read_audio("song.wav"), system, |
| clock="scheduled_stale_hold", deadline_ms=50, |
| ) |
| write("song.csv", times, frequencies) |
| ``` |
|
|
| Command-line example (replace `HUB_COMMIT` with a commit hash of this repository, or use `main`): |
|
|
| ```bash |
| hf download Frieve-A/jit-mpe weights/S-s1.safetensors weights/S-s1.calibration.json weights/head-selection-s1.json --revision HUB_COMMIT --local-dir hub-model |
| python -m jit_mpe.infer song.wav --weights hub-model/weights --system S --seed 1 --clock scheduled_stale_hold --deadline 50 --out song.csv |
| ``` |
|
|
| ### Input and output |
|
|
| - **Input:** any audio file. `read_audio` mixes it down to mono and resamples it to 44.1 kHz. |
| - **Output:** one row every 10 ms. A CSV row has the time in seconds, followed by the frequencies (Hz) of the pitches |
| that are sounding. An NPZ file has `times [Q]` and `frequencies [Q, 88]`; absent pitches are 0. |
| - **Pitch range:** MIDI 21–108 (A0–C8), with a fine frequency estimate for each active pitch. |
|
|
| ### Deadlines |
|
|
| A deadline D means that the answer for time q must be ready by q + D. With `scheduled_stale_hold`, the output for q |
| is the most recent frame that is ready by then, so there is always an answer, but it may be slightly old. |
|
|
| `S` has five delay heads that look 0, 20, 40, 80 and 160 ms ahead. For each deadline, the head-selection file says |
| which head to use; for 50 ms it is the 40 ms head. To process a whole file without a deadline, omit `clock` and |
| `deadline_ms` (Python) or `--clock` and `--deadline` (CLI). The `-bi` models only work this way, because they use |
| future audio. |
|
|
| `estimate` reads a complete file and reproduces what a real-time system would have output by each deadline. It is |
| not a live audio callback; for frame-by-frame streaming with `model.step`, see `bench/cpu.py` in the source code. |
|
|
| ## Training |
|
|
| The models were trained on GuitarSet, Slakh2100-redux, MusicNet, Schubert Winterreise, Cantoria, Dagstuhl ChoirSet, |
| vocadito and Groove MIDI. Versions, links and licenses are listed in [THIRD_PARTY.md](THIRD_PARTY.md). This repository |
| does not include any training audio, annotations, prepared features or external models. |
|
|
| Training has two stages: 10,400 steps at learning rate 0.002, then 3,000 steps at 0.0005 with an added onset loss |
| (batch size 16, 192-frame clips, 16 warm-up frames). Afterwards, the decision thresholds are fitted on a calibration |
| split and the delay heads are selected on a development set. The GitHub README gives the full commands for data |
| preparation, training and evaluation. `requirements-paper.txt` records the environment used for the paper |
| (Python 3.12, PyTorch 2.5.1); a CUDA build is not needed for CPU inference. |
|
|
| ## Results |
|
|
| The numbers below are copied from Table 4 of the paper ([arXiv v1](https://arxiv.org/html/2610.33113v1)); they were |
| not re-measured for this upload. They are F1 scores (%) for JIT-MPE with `scheduled_stale_hold` at a 50 ms deadline, |
| averaged per recording and over the three seeds, with pitches rounded to semitones. |
|
|
| | Held-out set | F1 (%) | |
| |---|---:| |
| | AAM | 82.6 | |
| | ChoraleBricks | 78.9 | |
| | BSED-real | 64.5 | |
| | BSED-synthetic | 77.0 | |
|
|
| Notes: |
|
|
| - The paper also reports 50-cent (continuous f0) scores and full-length scores. These use different settings and |
| should not be compared directly with the table above. |
| - ChoraleBricks is annotated with continuous f0; the semitone score above is given only so that all sets use the |
| same metric. |
| - The BSED recordings were not used in training, but some of the pieces also appear in the MusicNet training data. |
|
|
| See the paper for development-set results, comparisons with other systems, confidence intervals and scoring details. |
|
|
| ## Intended use and limitations |
|
|
| Intended for research on frame-level pitch estimation, pitch-display prototypes and experiments with output |
| deadlines. The models do not identify instruments and do not produce note events (onsets/offsets). |
|
|
| - The paper does not evaluate note-level transcription, singing voice or real popular-music recordings. |
| - Performance on other instruments or recording conditions has not been tested. |
| - The reported CPU timings depend on the hardware and do not include audio I/O, scheduling or display latency. |
| - Retraining with the released code will not give bit-identical weights, because these weights were trained before |
| deterministic training was added. The public vocadito split also differs from the one used in the paper (see the |
| determinism section of the GitHub README). To reproduce the paper's inference results, use these checkpoints. |
|
|
| ## License and citation |
|
|
| The code and these weights are released by the author under the [MIT license](https://huggingface.co/Frieve-A/jit-mpe/blob/main/LICENSE). This does not cover the |
| training datasets or external models, which keep their own licenses. In particular, the Schubert Winterreise |
| recordings have their own conditions, described in the original dataset release. |
|
|
| ```bibtex |
| @misc{kobayashi2026jitmpe, |
| title = {{JIT-MPE}: Just-in-Time Multi-Pitch Estimation under Output Deadlines}, |
| author = {Kobayashi, Yoshiyuki}, |
| year = {2026}, |
| eprint = {2610.33113}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.SD}, |
| doi = {10.48550/arXiv.2610.33113}, |
| url = {https://arxiv.org/abs/2610.33113} |
| } |
| ``` |
|
|
