README.md

---
license: cc-by-nc-4.0
library_name: muscriptor
tags:
- music
- music-transcription
- automatic-music-transcription
- amt
- audio-to-midi
- midi
- music-information-retrieval
- transformer
- pytorch
---

# MuScriptor — small (≈100M)

**MuScriptor** is an open-weight model for **general-purpose, multi-instrument automatic music transcription (AMT)**: it converts a music recording (any genre, multiple simultaneous instruments) into a stream of notes played. This repository hosts the **small** variant (≈100M parameters), the fastest and most lightweight checkpoint.

`muscriptor-small` is the smallest/fastest option, suitable for lower-resource settings. For higher quality use [`muscriptor-medium`](https://huggingface.co/MuScriptor/muscriptor-medium) (≈300M, good trade-off) or [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) (≈1.3B, best quality).

- Developed by [Mirelo](https://www.mirelo.ai/) x [kyutai](https://kyutai.org/)
- 📄 Paper: *MuScriptor: An Open Model for Multi-Instrument Music Transcription* — Rouard, Krause, Roebel, Simon-Gabriel, Défossez (2026). _<!-- TODO: add arXiv link once public; it will auto-cross-link on the Hub -->_
- 💻 Code: <https://github.com/muscriptor/muscriptor>
- 🔊 Audio samples: <https://muscriptor.github.io>

## Table of contents

- [Quickstart](#quickstart)
- [Model description](#model-description)
- [Model variants](#model-variants)
- [Intended uses & limitations](#intended-uses--limitations)
- [Instrument conditioning](#instrument-conditioning)
- [Training](#training)
- [Evaluation](#evaluation)
- [Citation](#citation)
- [License](#license)

## Quickstart

Install the `muscriptor` package (it uses `huggingface_hub` to fetch weights automatically):

```bash
pip install git+https://github.com/muscriptor/muscriptor.git
# TODO (PyPI release forthcoming: pip install muscriptor)
```

### Python

```python
from pathlib import Path
from muscriptor import TranscriptionModel

# "small" resolves to hf://MuScriptor/muscriptor-small and downloads on first use.
model = TranscriptionModel.load_model("small")

# Get a MIDI file directly:
Path("out.mid").write_bytes(model.transcribe_to_midi("audio.wav"))

# Or stream note events as they are transcribed:
for event in model.transcribe("audio.wav"):
    print(event)  # NoteStartEvent / NoteEndEvent / ProgressEvent
```

`load_model` accepts a size keyword (`"small"`/`"medium"`/`"large"`), a local `.safetensors` path, or an `hf://` / `https://` URL. Weights loaded by size keyword (or any `hf://` URL) are cached in the standard Hugging Face cache (`~/.cache/huggingface/hub`, configurable via `HF_HOME`); weights fetched from a plain `http(s)://` URL are cached under `~/.cache/muscriptor/`. Input audio can be WAV or any format `libsndfile` reads (mp3, flac, ogg, m4a, …); it is resampled to 16 kHz mono internally.

### CLI

```bash
muscriptor transcribe --model small audio.wav -o out.mid
```

## Model description

MuScriptor performs transcription by **autoregressively predicting a MIDI-like token sequence** given the mel-spectrogram of a short audio segment, following the sequence-to-sequence AMT paradigm (cf. MT3). It deliberately avoids complex architectural tweaks in favor of a simple, decoder-only Transformer.

- **Architecture:** decoder-only Transformer (this variant: `dim=768`, `num_heads=12`, `num_layers=14`).
- **Input:** raw waveform (16 kHz, mono) of a 5-second segment → mel-spectrogram (STFT `n_fft=2048`, hop 160 → 100 Hz frame rate, 512 mel bins). The spectrogram is projected to the model dimension and used as a prefix condition.
- **Output tokenization:** MT3-like note events; the 128 MIDI programs are mapped to **36 instrument subgroups** using the `MT3_FULL_PLUS` taxonomy. Decoding is greedy (argmax) by default, with optional classifier-free guidance (CFG).
- **Inference:** audio is processed in 5-second chunks; note events are emitted in temporal order. Optional **instrument conditioning** stabilizes predictions across chunk boundaries and lets you restrict/customize the transcription (see below).

**Note on the representation:** the tokenizer recovers onset/offset timing, pitch, and instrument, but **not velocity**. It also cannot represent two notes of the same pitch and instrument sounding at the same time. Drums are onset-only.

## Model variants

| Repo | Params | `dim` | heads | layers | Notes |
|---|---|---|---|---|---|
| [`muscriptor-small`](https://huggingface.co/MuScriptor/muscriptor-small) | ≈100M | 768 | 12 | 14 | **this model** · smallest / fastest |
| [`muscriptor-medium`](https://huggingface.co/MuScriptor/muscriptor-medium) | ≈300M | 1024 | 16 | 24 | good trade-off |
| [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) | ≈1.3B | 1536 | 24 | 48 | best quality |

All variants share the same input pipeline, tokenizer, and training recipe; they differ only in latent dimension, attention heads, and depth.

## Intended uses & limitations

**Intended uses**
- General-purpose transcription of real, multi-instrument music across genres (classical → heavy metal) into MIDI.
- A building block for music information retrieval (chord/key recognition), musicological analysis, generative-modeling data pipelines, and tools for musicians.
- Lower-resource / latency-sensitive settings where the larger variants are too heavy.

**Out of scope / use with care**
- Not a substitute for a hand-annotated score; expect errors, especially on dense mixes, unusual timbres, and heavily processed audio.
- Velocity/dynamics are **not** produced (see note above).
- Onset/offset precision is lower for some styles (e.g. choral music), and exact offsets are inherently harder than onsets.
- Being the smallest variant, it trades transcription accuracy for speed and footprint relative to `medium`/`large`.

**Limitations & biases**
- Training data skews toward pop and Western classical music, and the instrument distribution is long-tailed (piano/guitar/bass/drums are most frequent). Rare instruments and underrepresented genres may be transcribed less reliably.
- The fixed `MT3_FULL_PLUS` 36-group instrument taxonomy limits instrument granularity.
- Simultaneous same-pitch/same-instrument notes cannot be represented by the tokenizer.

## Instrument conditioning

The model can be told which instrument groups are present in the track. Supplying the correct set improves quantitative scores and produces more coherent instrument assignments across segments.

```python
from muscriptor.tokenizer.mt3 import MT3_FULL_PLUS_GROUP_NAMES

# `instrument_group` is a space-separated string of MT3_FULL_PLUS group IDs.
# Convert readable group names to IDs:
names = ["acoustic_piano", "acoustic_guitar", "acoustic_bass"]
instrument_group = " ".join(str(MT3_FULL_PLUS_GROUP_NAMES[n]) for n in names)  # -> "0 4 7"

# Only expect piano, acoustic guitar and bass in this track:
model.transcribe_to_midi("audio.wav", instrument_group=instrument_group)
```

```bash
muscriptor transcribe --model small --instruments "acoustic_piano,acoustic_guitar,acoustic_bass" audio.wav -o out.mid
muscriptor list-instruments   # show all available group names
```

## Evaluation

Metrics are instrument-agnostic F1 scores computed with [`mir_eval`](https://github.com/craffel/mir_eval) on `D_Test`, the authors' held-out test set of 372 multi-instrument tracks.

**Model-size comparison** (F1 ↑; from the paper's scaling study, models trained on `D_Real` only, CFG = 2):

| Variant | Params | Onset | Frame | Offset | Drums | Multi |
|---|---|---|---|---|---|---|
| **`muscriptor-small`** | **100M** | **51.2** | **67.2** | **38.7** | **41.5** | **38.2** |
| `muscriptor-medium` | 300M | 52.4 | 68.0 | 40.3 | 42.0 | 39.7 |
| `muscriptor-large` | 1.3B | 53.2 | 68.7 | 41.0 | 42.5 | 40.5 |

These numbers come from the model-size ablation, which trains on real audio **only**. The **released checkpoints additionally use synthetic pre-training and RL post-training**, which improve real-world quality substantially beyond these figures. See [`muscriptor-large`](https://huggingface.co/MuScriptor/muscriptor-large) and the paper for per-dataset results.

## Citation

```bibtex
@inproceedings{muscriptor2026,
  title     = {MuScriptor: An Open Model for Multi-Instrument Music Transcription},
  author    = {Rouard, Simon and Krause, Michael and Roebel, Axel and
               Simon-Gabriel, Carl-Johann and D{\'e}fossez, Alexandre},
  year      = {2026},
  note      = {Kyutai, Mirelo AI, IRCAM}
}
```

<!-- TODO: replace with the final published citation (venue / arXiv id) once available. -->

## License

Code released under the [MIT License](https://github.com/muscriptor/muscriptor/blob/main/LICENSE). Weights released under CC-BY-NC.