--- license: apache-2.0 library_name: transformers pipeline_tag: text-generation tags: - symbolic-music - music-generation - musicxml - midi - remi - bar-major --- # Clef-0.2B A ~0.2B-parameter decoder-only Transformer for **symbolic music generation** (MusicXML / MIDI). Given a short conditioning header (genre / ensemble / tempo / key / meter / instrument plan) it generates a multi-track score, and it also supports **CONTINUE** (extend a given prefix) and **INFILL** (fill masked bars). The model was pretrained from scratch on symbolic scores and then instruction-tuned on those three tasks. This is the instruction-tuned checkpoint. ## Model summary | | | |---|---| | Architecture | decoder-only Transformer, RoPE, RMSNorm, SwiGLU, GQA | | Params | 201,398,272 (~0.2B) | | Layers / d_model | 17 / 1024 | | Attention | 16 query heads : 4 KV heads (GQA), FFN 2816 | | Context length | **4096** tokens max (trained on 2048-token windows) | | Tokenizer | `tokenizer_v2` — bar-major REMI+, vocab **9507**, 24 TPQN | | Precision | trained in fp8 (torchao) + `torch.compile`; weights ship bf16 | ### Weights format Inference weights are provided as **`model.safetensors`** in **Llama (`LlamaForCausalLM`) layout** with a matching `config.json`, so they load with `transformers` / `vLLM` directly. This layout was numerically verified against the native training checkpoint (greedy decode matches token-for-token). Note the **tokenizer is custom** (bar-major REMI+, in `tokenizer/`) — it is *not* an `AutoTokenizer`, so encoding/decoding to MusicXML/MIDI uses this project's pipeline, not `AutoTokenizer.from_pretrained`. ### Tokenizer / codec The tokenizer is **bar-major REMI+**: tracks are interleaved bar by bar so all instruments advance together, which gives the model an explicit notion of simultaneity and helps parts **stop together** instead of one voice trailing off. A generation begins with a native conditioning header: ``` BOS [GENRE] [ENSEMBLE] TEMPO KEY METER {TRACK|DRUM}* (BAR ...)* EOS ``` Drums are encoded by their **true General MIDI percussion number** (resolved from `/midi-instrument/midi-unpitched`), and note velocity / dynamics are preserved. Because drums are stored as real GM voices, decoded output renders correct kick / snare / hi-hat directly with no display→GM remap. ## Prompt example A generation is conditioned by a header of special tokens; the model then produces the bars after ``. These are the exact prompts used for the sample files below. Rock band — distortion + overdrive guitar, electric bass, drums; A minor, 130 BPM: ``` ``` Piano concerto — piano + string ensemble + cello; E♭ major, 96 BPM: ``` ``` - `` picks a General MIDI program (0 = piano, 30 = distortion guitar, 33 = electric bass, 48 = string ensemble, 42 = cello, …). - `` adds a percussion lane. - `` uses `k` = signed sharps(+) / flats(−); `` = BPM; `` = time signature. ### Conditioning vocabulary - **GENRE** `` (9): `classical` · `popular` · `jazz` · `folk` · `electronic` · `soundtrack` · `world` · `religious` · `unknown` - **ENSEMBLE** `` (7): `solo` · `duo` · `band` · `chamber` · `orchestral` · `choir` · `other` - **TRACK** ``: any General MIDI program, `n` = 0–127; plus `` for a percussion lane. - **KEY** ``: `k` = −7…+7 (sharps/flats), `mode` ∈ `major` · `minor` · `dorian` · `phrygian` · `lydian` · `mixolydian` · `none`. - **TEMPO** ``: integer BPM (5–999). - **METER** ``: time signature (e.g. `4/4`, `3/4`, `6/8`). (The genre/ensemble labels are coarse; `unknown` / `other` are valid and common.) ## Samples `samples/promo_rock.mid` and `samples/promo_piano_concerto.mid` are ~3-minute pieces generated from the two prompts above (best-of-N seeds, CONTINUE-stitched past the context window). Render with any General MIDI SoundFont. ## Training - **Pretraining:** from scratch, 3 epochs (13,821 steps), global batch 256, fp8 + `torch.compile`. Data: ~1.18M bar-major windows built from the public sources listed below, with ×12 transpose augmentation. - **Instruction tuning (this checkpoint):** GENERATE / CONTINUE / INFILL tasks mixed with in-key curation (in-key ≥ 0.88) and a base-replay set to limit forgetting. 760 steps (2 epochs), best **val loss 0.3088 @ step 299** (adopted). Hardware: 1× RTX PRO 6000 Blackwell (96 GB), ~80k tok/s, ~45 min. ## Evaluation Automatic metrics on held-out generation prompts (2 seeds). Values are absolute for this model; real human-composed scores are given only as a rough anchor. | metric | Clef-0.2B | real scores (ref) | |---|--:|--:| | in-key ratio ↑ | 0.81 | ~0.96 | | dissonance ↓ | 0.29 | ~0.15 | | co-termination spread ↓ | 0.14 | — | | early part death ↓ | 0.10 | — | Task success (model emits ≥ 1 new valid bar for the task): | task | success | |---|--:| | CONTINUE | 12/12 | | INFILL | 12/12 | - **in-key ratio** — fraction of notes inside the conditioned key. - **dissonance** — ratio of dissonant simultaneous intervals. - **co-termination spread / early part death** — how tightly instruments end together vs. voices dropping out early (lower is better). ## How to generate Generation runs in this project's pipeline (bar-major decode → MusicXML/MIDI), then any General MIDI SoundFont (e.g. MuseScore) renders audio. **Recommended defaults:** `temperature 0.7`, `top_k 24`, `repetition_penalty ~1.12`. This is the setting used for the sample pieces and is a good starting point. Optional (supported when decoding with vLLM / transformers): `top_p 0.95` and `presence_penalty 0.15` for extra variety. ⚠️ **Do not use frequency_penalty** — it destabilizes the structural tokens (BAR / TRK / PLAN). For pieces longer than the context window, continue past it with a slightly lower **continuation temperature (~0.76)** to soften seams. Optionally post-process drums with a `sanitize_drums` step that snaps rare out-of-kit hits (e.g. GM 71/72) back to a standard kit. ## Limitations - **Context is 4096 tokens (~35 bars).** Longer pieces are produced by stitching CONTINUE windows, which can cause a slight mood shift at the seam. A long-context retrain (or RoPE scaling at inference) would fix this properly. - **Dissonance** stays somewhat above real-score levels. - Research artifact; quality is not guaranteed for production use. ## Training data Built from the following public symbolic-music sources: - **PDMX** — https://github.com/pnlong/PDMX - **KernScores / Humdrum** — https://github.com/humdrum-tools/humdrum-data · https://kern.humdrum.org/ - **OpenScore Lieder** — https://github.com/OpenScore/Lieder - **The Session** — https://github.com/adactio/TheSession-data ## License Released under **Apache-2.0**.