SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton

SymphonyGen is a 3D hierarchical framework for contemporary cinematic orchestration, accepted at ISMIR 2026. It decomposes symphonic scores along the Bar, Track, and Event axes with a cascading decoder architecture, conditions generation on a beat-quantized multi-voice harmony skeleton ("short-score" conditioning), refines the model with GRPO using a cross-modal audio-perceptual reward (CLaMP 3 audio embeddings), and suppresses tonal clashes at inference time with dissonance-averse sampling.

• 📄 Paper: SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton (ISMIR 2026)
• 💻 Code: github.com/symphonygen/symphonygen
• 🌐 Audio demo page: symphonygen.github.io (listening-test excerpts for composition and re-orchestration, plus baselines)

Released checkpoints

All checkpoints are packed: each .pt bundles the model config together with the weights, and is loaded by ModelBase.from_pretrained in the codebase.

File	Model	Role
stage_one_pretrained.pt	1D harmony skeleton model (12 layers, hidden 768, 87M)	Samples harmony skeletons
stage_two_pretrained.pt	3D symphony model (33 layers, hidden 512, 124M, 2-stream cross-attention)	Pretrained on the SymphonyNet Dataset; starting point for GRPO / finetuning
grpo_clamp_epoch_10.pt	3D symphony model ("reinforced")	GRPO with the pure CLaMP 3 audio reward (paper round-1 subjective test)
grpo_clamp+track_epoch_6.pt	3D symphony model ("reinforced+track")	GRPO with the composite CLaMP 3 + track density reward (paper round-2 subjective test)

Recommended dissonance-averse sampling settings: (λ_hn, λ_nn) = (1, 10); use register decay (--register_decay 1) with reinforced+track and --register_decay 0 with reinforced.

Usage

Generation is run through the codebase (see its README for full setup):

git clone https://github.com/symphonygen/symphonygen
cd symphonygen && pip install -r requirements.txt
export PYTHONPATH=$(pwd)

# Download the checkpoints into $ASSET_DIR (default: asset/)
hf download SymphonyGen/SymphonyGen --local-dir asset/

# Generate a harmony skeleton, then orchestrate it
python arch/harmo/generator.py asset/stage_one_pretrained.pt --batch_size 4 --save_dir harmony_out
python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt harmony_out \
    --group_size 1 --save_dir songs --forbid_piano

# Or re-orchestrate any MIDI by analyzing its harmony skeleton
python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt your_song.mid \
    --analyze_harmo --save_dir songs

The harmony skeleton may also be user-written (e.g. a few chords played on a MIDI keyboard, exported as MIDI) — pass it as the conditioning MIDI without --analyze_harmo.

Training

• Data: the SymphonyNet Dataset (728 classical + 45,632 contemporary MIDI files), 90/10 train/validation split.
• Pretraining: four NVIDIA H800 GPUs for one day, AdamW, lr 1e-4 with cosine annealing.
• GRPO: single GPU, lr 4e-5, group size 32 over 16 skeletons per epoch; reward = CLaMP 3 audio-embedding cosine similarity against the centroid of a small cinematic reference set (rendered with MuseScore 3), optionally + 0.2 × track-density term.

See the paper and the repository's DIFF.md for implementation details that go beyond the paper's description.

Evaluation

Objective metrics (CLaMP similarity, dissonance, track density, harmony precision/recall, melodic movement/ornament) and two rounds of subjective listening tests are reported in the paper; reproduction scripts live under Results/ in the codebase.

Reproducibility disclaimer: the objective metrics depend on the exact implementation of the harmony analysis and of the harmony-skeleton filters (see DIFF.md in the code repository). As a result, you might see reproduced objective evaluation numbers that differ from the paper's tables. The trends, however, should be the same.

Intended use and limitations

SymphonyGen is intended as a collaborative aid for composers, not as an autonomous replacement. It is oriented toward contemporary cinematic scoring through its data, reward, and hyperparameters. Generative outputs may inadvertently resemble existing works; users should not assume all outputs are fully original without verification.

Citation

@inproceedings{symphonygen2026,
  title     = {SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton},
  author    = {He, Xuzheng and Nan, Nan and Wang, Zhilin and Kang, Ziyue and Mo, Zhuoru and Li, Ao and Pan, Yu and Li, Xiaobing and Yu, Feng and Guan, Xiaohong},
  booktitle = {Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR)},
  year      = {2026}
}