abc-gpt

A small GPT-2 (26.8M parameters) trained from scratch to write Irish folk tunes in ABC notation. It was pretrained on raw ABC text, finetuned to follow a control code that sets the structure of the tune (number of sections, bars per section, similarity between sections), then trained with GRPO on rewards computed with music21.

It is the companion model of a presentation on the Hugging Face ecosystem: each stage uses a library of that stack (tokenizers, datatrove, transformers, trl, gradio). Code: github.com/matteospanio/abc-gpt.

Files

Path What it is
/ (root) the final model, after GRPO: what from_pretrained("matteospanio/abc-gpt") loads
sft/ the model after supervised finetuning, before GRPO
pretrain/ the model after pretraining only: it writes ABC but does not read the control code
abc-gpt-f16.gguf the final model in GGUF (F16), for llama.cpp and compatible runtimes

Usage

The prompt is the control code in an ABC comment line, followed by the tune header; the model writes the body of the tune and stops with its end-of-text token.

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("matteospanio/abc-gpt")
model = AutoModelForCausalLM.from_pretrained("matteospanio/abc-gpt")  # subfolder="sft" or "pretrain" for the earlier stages

prompt = "%S:2 B:8 E:5 B:8\nX:1\nL:1/8\nM:6/8\nK:D\n"  # a jig in D: 2 sections of 8 bars, the second 50% similar to the first
inputs = tokenizer(prompt, return_tensors="pt")
out = model.generate(**inputs, do_sample=True, temperature=1.0, max_new_tokens=512,
                     pad_token_id=tokenizer.pad_token_id)
print(prompt + tokenizer.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Use sampling: with greedy decoding a prompt always gives the same tune, and a repetitive one (in a test, a 6/8 tune of 16 bars had only 5 different ones). Paste the result in an ABC player (for example abcjs or EasyABC) to see and hear it.

The GGUF file loads in transformers too (with the gguf package installed):

model = AutoModelForCausalLM.from_pretrained("matteospanio/abc-gpt", gguf_file="abc-gpt-f16.gguf")
tokenizer = AutoTokenizer.from_pretrained("matteospanio/abc-gpt", gguf_file="abc-gpt-f16.gguf")

llama.cpp

hf download matteospanio/abc-gpt abc-gpt-f16.gguf --local-dir .
llama completion -m abc-gpt-f16.gguf -p '%S:2 B:8 E:5 B:8\nX:1\nL:1/8\nM:6/8\nK:D\n' -e \
  -n 512 --temp 1.0 --top-k 0 --top-p 1.0 --min-p 0.0

Use completion (plain text continuation), not the chat interface. Give the prompt with -p and -e (escapes): with -f llama.cpp drops the final newline of the file, and the model was trained on headers that end with one. --top-k 0 --top-p 1.0 --min-p 0.0 turn off llama.cpp's default filters and give the plain sampling of the evaluation below. Tested with llama.cpp build 12182: the prompt is split into the same tokens as with transformers, and the tunes end at the end-of-text token.

Prompt format

Line Meaning
%S:2 B:8 E:5 B:8 control code: S: sections, then for each section one E: (0-10, similarity to each earlier section) and B: (bars)
X:1 tune number
L:1/8 unit note length (the model was trained with the headers of IrishMAN, mostly 1/8)
M:6/8 meter: 6/8 jig, 4/4 reel, 3/4, 2/4, 9/8 slip jig, ...
K:D key and mode: D, G, Emin, Amin, Dmix, Ador, ...

The control code follows IrishMAN: sections end at [|, ||, |], |:, :: or :|, B counts the bars of a section, and E is 1 - levenshtein(c, p) / max(|c|, |p|) between section c and an earlier section p, in 11 levels.

Speed

On a laptop CPU (Intel Core Ultra 5 226V, 4 threads), a tune of about 100 tokens takes:

Runtime Tokens per second Time per tune
transformers (float32) about 180 about 0.4 s
llama.cpp (abc-gpt-f16.gguf) about 770 about 0.1 s

With transformers, more threads are slower for a model this small: 8 threads give about 110 tokens per second, so set torch.set_num_threads(4).

Model

GPT-2 architecture (GPT2LMHeadModel), trained from random weights.

Layers Γ— width (heads) 8 Γ— 512 (8)
Parameters 26.8M
Context 1,024 positions; trained on sequences of at most 512 tokens
Tokenizer byte-level BPE trained on the IrishMAN tunes, 2,048 tokens; `<
Weights float32 (model.safetensors, 107 MB); F16 in the GGUF file (55 MB)

Training

Data: IrishMAN, 214,122 training and 2,162 validation tunes in ABC notation, with titles and lyrics removed and a control code computed for each tune. All three stages ran with TRL, in bf16.

Stage Data Settings Result
Pretraining (trl sft) 214,122 training tunes, raw ABC text 3 epochs, 20,076 steps, batch 32, lr 6e-4 cosine, 500 warmup steps, weight decay 0.1 validation loss 1.610 (step 2,000) β†’ 1.010
Supervised finetuning (trl sft) 214,107 tunes (those with a body) as prompt-completion pairs: control code and header β†’ body; loss on the completion only 1 epoch, 6,691 steps, batch 32, lr 5e-5 cosine, 200 warmup steps validation loss 1.068 (step 1,000) β†’ 0.994
GRPO (trl grpo) 5,321 prompts (2.5% of the training tunes, after dropping unreadable and too-long tunes) 500 steps, 8 tunes per prompt, batch 8 Γ— 4 accumulation, lr 1e-5, no KL penalty (beta: 0), temperature 1.0 mean reward 3.27 β†’ 3.36 (first and last 50 steps)

GRPO rewards, each between 0 and 1:

Reward Weight Measures First 50 steps Last 50 steps
syntax 1.0 music21 parses the whole tune without warnings and finds notes 0.968 0.992
meter 1.0 share of bars whose length matches the time signature 0.906 0.955
note_order 0.5 share of notes written in the canonical ABC order (accidental, pitch, octave, length) 0.998 0.999
structure 1.0 how close the tune's own control code is to the one in the prompt 0.894 0.908

Framework versions: TRL 1.17.1, Transformers 5.22.2, PyTorch 2.15.0, Datasets 5.2.0, Tokenizers 0.23.3.

Evaluation

100 tunes per checkpoint, from the same prompts: the first 100 of 176 prompts drawn from the IrishMAN validation split (10% of the validation tunes that pass the data filter, seed 42), sampled at temperature 1.0 with python -m abcgpt.evaluate on CPU. Each reward column is a mean between 0 and 1; the last column counts the tunes that music21 exports to MusicXML.

Checkpoint syntax meter note_order structure MusicXML
pretrain/ 0.930 0.292 0.996 0.435 93
sft/ 1.000 0.941 1.000 0.898 100
root (GRPO) 1.000 0.948 0.999 0.902 100

The control code of each generated tune, compared with the one in its prompt:

Checkpoint same number of sections same sections and bars identical code
pretrain/ 27% 1% 0%
sft/ 88% 42% 18%
root (GRPO) 90% 44% 14%

Finetuning makes the difference: the pretrained model never saw the control-code line in training. GRPO adds little on these prompts: the SFT model already scored close to 1 on syntax and note_order, and its gains on meter (+0.007) and structure (+0.004) are smaller than the standard error of these means over 100 tunes (about 0.015-0.019).

Limitations

  • The model knows only the tunes of IrishMAN: Irish and other traditional dance tunes, single melody line, a few chord symbols. It does not write titles, lyrics or other styles.
  • The rewards check that the ABC is readable and that bars and sections add up. They say nothing about whether a tune is good, and nobody has listened to the generated tunes systematically.
  • The control code is followed approximately: in the evaluation above the number of sections was right in 90% of the tunes, the bars of every section in 44%, the whole code in 14%.
  • A model trained for several epochs on 214k tunes may reproduce tunes of the training set; this has not been measured.
  • IrishMAN is MIT-licensed, but its card says it is "for research use only and not for commercial purposes". Keep that in mind before using generated tunes commercially.

Citation

The dataset and the control codes come from TunesFormer:

@inproceedings{DBLP:conf/hcmir/WuLY023,
  author    = {Shangda Wu and Xiaobing Li and Feng Yu and Maosong Sun},
  title     = {TunesFormer: Forming Irish Tunes with Control Codes by Bar Patching},
  booktitle = {Proceedings of the 2nd Workshop on Human-Centric Music Information Retrieval 2023
               co-located with the 24th International Society for Music Information Retrieval
               Conference (ISMIR 2023), Milan, Italy, November 10, 2023},
  series    = {CEUR Workshop Proceedings},
  volume    = {3528},
  publisher = {CEUR-WS.org},
  year      = {2023},
  url       = {https://ceur-ws.org/Vol-3528/paper1.pdf}
}
Downloads last month
43
Safetensors
Model size
26.8M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train matteospanio/abc-gpt