HarmonyRL

Symbolic piano music generation: supervised pretraining on MAESTRO, then PPO fine-tuning against musical reward functions.

Code: https://github.com/SupratikB23/HarmonyRL

Files

File	What it is
checkpoints/transformer_supervised.pt	Supervised pretrained model, epoch 12
checkpoints/transformer_rl.pt	PPO fine-tuned, 3000 iterations, includes the critic

Each checkpoint carries its own model config, so it rebuilds the exact architecture with no hardcoded dimensions.

Model

25.3M-parameter pre-norm decoder-only Transformer. d_model 512, 8 layers, 8 heads, feed-forward 2048, context 2048. Rotary embeddings applied to queries and keys inside each attention head, causal masking, KV cache for generation. Vocabulary is 172 REMI-style event tokens: bar, position, pitch, velocity, duration. A note is four tokens, and notes sharing a position stay simultaneous, so polyphony is preserved.

Training

Trained on MAESTRO v3.0.0, 1276 MIDI files, 28.5M tokens, 53,594 chunks of 1024 tokens at 50 percent stride. Train and validation are split by recording group so movements of one performance cannot straddle the split.

Validation perplexity	3.16 (random baseline 172)
Stopped at	epoch 12, early stopping
PPO	3000 iterations

PPO uses token-level GAE with a value head, and a KL penalty against a frozen copy of the pretrained model. The KL anchor is what stops the policy collapsing onto degenerate output that games the reward. Reward is five symbolic terms: harmony, scale, rhythm, diversity, density.

Results

Generated samples hold 11 to 37 simultaneous note onsets, use 17 distinct velocity levels, score 0.98 to 1.00 on distinct 4-gram ratio, and have a longest repeated-note run of 3 to 4. No degenerate loops.

Limitations

Piano only, single instrument, trained on classical performance so it will not generalize to other genres. Tonality is the weak axis: scale adherence sits at 0.65 to 0.74 and pitch spreads across the full 88-key range, so output wanders between registers rather than staying in one. Generations are short, typically under a minute, with no long-range structure such as themes or repeats. The reward functions are hand-written proxies, not learned from human preference.

Usage

from harmonyrl.utils import load_model
from harmonyrl.midi_utils import EOS, tokens_to_midi

model, meta = load_model("checkpoints/transformer_rl.pt", device="cuda")
seq = model.sample(batch_size=1, max_new_tokens=1024, temperature=0.95,
                   top_p=0.95, device="cuda")

row = seq[0].tolist()
tokens = row[: row.index(EOS) + 1] if EOS in row else row
tokens_to_midi(tokens).write("out.mid")