README.md

---
license: mit
tags:
- music
- music-generation
- symbolic-music
- midi
- orchestral-music
- transformer
- grpo
- reinforcement-learning
library_name: pytorch
---

# SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

[![arXiv](https://img.shields.io/badge/arXiv-2604.25498-b31b1b.svg)](https://arxiv.org/abs/2604.25498)
[![Code](https://img.shields.io/badge/GitHub-Code-black?logo=github)](https://github.com/symphonygen/symphonygen)
[![Demo Page](https://img.shields.io/badge/%F0%9F%8C%90%20Demo-symphonygen.github.io-green)](https://symphonygen.github.io)

SymphonyGen is a 3D hierarchical framework for contemporary cinematic orchestration, accepted at **ISMIR 2026**. It decomposes symphonic scores along the Bar, Track, and Event axes with a cascading decoder architecture, conditions generation on a beat-quantized multi-voice **harmony skeleton** ("short-score" conditioning), refines the model with **GRPO** using a cross-modal audio-perceptual reward (CLaMP 3 audio embeddings), and suppresses tonal clashes at inference time with **dissonance-averse sampling**.

- 📄 **Paper:** [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://arxiv.org/abs/2604.25498) (ISMIR 2026)
- 💻 **Code:** [github.com/symphonygen/symphonygen](https://github.com/symphonygen/symphonygen)
- 🌐 **Audio demo page:** [symphonygen.github.io](https://symphonygen.github.io) (listening-test excerpts for composition and re-orchestration, plus baselines)

## Released checkpoints

All checkpoints are **packed**: each `.pt` bundles the model config together with the weights, and is loaded by `ModelBase.from_pretrained` in the codebase.

| File | Model | Role |
| --- | --- | --- |
| `stage_one_pretrained.pt` | 1D harmony skeleton model (12 layers, hidden 768, 87M) | Samples harmony skeletons |
| `stage_two_pretrained.pt` | 3D symphony model (33 layers, hidden 512, 124M, 2-stream cross-attention) | Pretrained on the SymphonyNet Dataset; starting point for GRPO / finetuning |
| `grpo_clamp_epoch_10.pt` | 3D symphony model ("reinforced") | GRPO with the pure CLaMP 3 audio reward (paper round-1 subjective test) |
| `grpo_clamp+track_epoch_6.pt` | 3D symphony model ("reinforced+track") | GRPO with the composite CLaMP 3 + track density reward (paper round-2 subjective test) |

Recommended dissonance-averse sampling settings: `(λ_hn, λ_nn) = (1, 10)`; use register decay (`--register_decay 1`) with `reinforced+track` and `--register_decay 0` with `reinforced`.

## Usage

Generation is run through the [codebase](https://github.com/symphonygen/symphonygen) (see its README for full setup):

```bash
git clone https://github.com/symphonygen/symphonygen
cd symphonygen && pip install -r requirements.txt
export PYTHONPATH=$(pwd)

# Download the checkpoints into $ASSET_DIR (default: asset/)
hf download SymphonyGen/SymphonyGen --local-dir asset/

# Generate a harmony skeleton, then orchestrate it
python arch/harmo/generator.py asset/stage_one_pretrained.pt --batch_size 4 --save_dir harmony_out
python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt harmony_out \
    --group_size 1 --save_dir songs --forbid_piano

# Or re-orchestrate any MIDI by analyzing its harmony skeleton
python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt your_song.mid \
    --analyze_harmo --save_dir songs
```

The harmony skeleton may also be **user-written** (e.g. a few chords played on a MIDI keyboard, exported as MIDI) — pass it as the conditioning MIDI without `--analyze_harmo`.

## Training

- **Data:** the [SymphonyNet Dataset](https://symphonynet.github.io/) (728 classical + 45,632 contemporary MIDI files), 90/10 train/validation split.
- **Pretraining:** four NVIDIA H800 GPUs for one day, AdamW, lr 1e-4 with cosine annealing.
- **GRPO:** single GPU, lr 4e-5, group size 32 over 16 skeletons per epoch; reward = CLaMP 3 audio-embedding cosine similarity against the centroid of a small cinematic reference set (rendered with MuseScore 3), optionally + 0.2 × track-density term.

See the paper and the repository's `DIFF.md` for implementation details that go beyond the paper's description.

## Evaluation

Objective metrics (CLaMP similarity, dissonance, track density, harmony precision/recall, melodic movement/ornament) and two rounds of subjective listening tests are reported in the paper; reproduction scripts live under `Results/` in the codebase.

> **Reproducibility disclaimer:** the objective metrics depend on the exact implementation of the harmony analysis and of the harmony-skeleton filters (see `DIFF.md` in the code repository). As a result, you might see reproduced objective evaluation numbers that differ from the paper's tables. The trends, however, should be the same.

## Intended use and limitations

SymphonyGen is intended as a collaborative aid for composers, not as an autonomous replacement. It is oriented toward contemporary cinematic scoring through its data, reward, and hyperparameters. Generative outputs may inadvertently resemble existing works; users should not assume all outputs are fully original without verification.

## Citation

```bibtex
@inproceedings{symphonygen2026,
  title     = {SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton},
  author    = {He, Xuzheng and Nan, Nan and Wang, Zhilin and Kang, Ziyue and Mo, Zhuoru and Li, Ao and Pan, Yu and Li, Xiaobing and Yu, Feng and Guan, Xiaohong},
  booktitle = {Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR)},
  year      = {2026}
}
```