| --- |
| license: mit |
| tags: |
| - music |
| - music-generation |
| - midi |
| - pytorch |
| - onnx |
| - custom |
| datasets: |
| - drengskapur/midi-classical-music |
| pipeline_tag: text-generation |
| --- |
| |
| # Music Generation LLM |
|
|
| A LLaMA-style Transformer model for symbolic music generation, trained on MIDI data. |
|
|
| ## Architecture |
|
|
| - **Model**: LLaMA-style Transformer with RoPE, GQA, SwiGLU, RMSNorm |
| - **Tokenizer**: REMI (REvamped MIDI-derived) — SOTA for symbolic music |
| - **Dataset**: `drengskapur/midi-classical-music` — 4,796 classical MIDI files (~50MB) |
| - **Training**: AdamW + Cosine LR warmup + AMP + Gradient Checkpointing |
|
|
| ## Key Features |
|
|
| - **Memory Efficient**: Grouped Query Attention, gradient checkpointing, mixed precision |
| - **OOM Safe**: Conservative batch sizes, AMP, lazy data loading |
| - **SOTA Techniques**: RoPE, SwiGLU, RMSNorm, KV-cache, top-p/top-k sampling |
|
|
| ## Quick Start |
|
|
| ```bash |
| # Install dependencies |
| pip install -r requirements.txt |
| |
| # Train + Generate (default) |
| python3 -m src.s00_main train+generate |
| |
| # Train only |
| python3 -m src.s00_main train --epochs 20 --batch-size 4 |
| |
| # Generate from checkpoint |
| python3 -m src.s00_main generate --temperature 0.85 |
| ``` |
|
|
| ## Project Structure |
|
|
| ``` |
| music_gen_llm/ |
| ├── src/ |
| │ ├── s00_main.py # Entry point — orchestrates pipeline |
| │ ├── s01_config.py # All configuration dataclasses |
| │ ├── s02_tokenizer.py # REMI MIDI tokenizer |
| │ ├── s03_dataset.py # Data download + tokenization + DataLoader |
| │ ├── s04_model.py # MusicTransformer (LLaMA-style) |
| │ ├── s05_trainer.py # Training loop with AMP + checkpointing |
| │ ├── s06_generator.py # Autoregressive generation with KV-cache |
| │ └── s07_utils.py # Logging, memory monitoring, seeding |
| ├── tests/ |
| │ └── test_pipeline.py # 7 unit tests covering all components |
| ├── scripts/ |
| │ ├── download_data.sh # Dataset setup |
| │ ├── train.sh # Training launcher |
| │ └── generate.sh # Generation launcher |
| ├── docs/ |
| │ ├── README.md # This file |
| │ ├── HLD.md # High-Level Design |
| │ ├── LLD.md # Low-Level Design |
| │ └── flow_diagram.drawio # Execution flow diagram |
| ├── data/ # Downloaded MIDI + tokenized cache |
| ├── checkpoints/ # Saved model weights |
| ├── output/ # Generated MIDI files |
| ├── requirements.txt |
| ├── Dockerfile |
| └── .gitignore |
| ``` |
|
|
| ## Execution Flow |
|
|
| ``` |
| s00_main.py → s01_config.py → s02_tokenizer.py → s03_dataset.py → s04_model.py → s05_trainer.py → s06_generator.py |
| │ │ │ │ │ │ │ |
| Entry point Load configs Init tokenizer Download & tokenize Build model Train loop Generate MIDI |
| ``` |
|
|
| ## Model Specifications |
|
|
| | Parameter | Value | |
| |---------------|-------------------------------------| |
| | Dim | 256 | |
| | Layers | 6 | |
| | Heads | 8 (Q) / 4 (KV) — GQA | |
| | Hidden (FFN) | 448 (SwiGLU) | |
| | Max Seq Len | 1024 | |
| | Vocab Size | 485 (REMI tokens) | |
| | Parameters | ~5M | |
| | Precision | BF16/FP16 (AMP) | |
|
|
| ## Algorithms Used |
|
|
| 1. **Rotary Position Embeddings (RoPE)** — Su et al. 2021 |
| 2. **Grouped Query Attention (GQA)** — Ainslie et al. 2023, from LLaMA-2 |
| 3. **SwiGLU Activation** — Shazeer 2020, from LLaMA |
| 4. **RMS Layer Normalization** — Zhang & Sennrich 2019 |
| 5. **REMI Tokenization** — Huang & Yang 2020 |
| 6. **Cosine Annealing with Warmup** — Loshchilov & Hutter 2017 |
| 7. **Gradient Checkpointing** — Chen et al. 2016 |
| 8. **KV-Cache** — Standard for efficient autoregressive decoding |
| 9. **Nucleus (Top-p) + Top-k Sampling** — Holtzman et al. 2020 |
| 10. **Repetition Penalty** — Keskar et al. 2019 |
|
|
