File size: 4,168 Bytes
bbc981d 30e9297 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 | ---
license: mit
tags:
- music
- music-generation
- midi
- pytorch
- onnx
- custom
datasets:
- drengskapur/midi-classical-music
pipeline_tag: text-generation
---
# Music Generation LLM
A LLaMA-style Transformer model for symbolic music generation, trained on MIDI data.
## Architecture
- **Model**: LLaMA-style Transformer with RoPE, GQA, SwiGLU, RMSNorm
- **Tokenizer**: REMI (REvamped MIDI-derived) — SOTA for symbolic music
- **Dataset**: `drengskapur/midi-classical-music` — 4,796 classical MIDI files (~50MB)
- **Training**: AdamW + Cosine LR warmup + AMP + Gradient Checkpointing
## Key Features
- **Memory Efficient**: Grouped Query Attention, gradient checkpointing, mixed precision
- **OOM Safe**: Conservative batch sizes, AMP, lazy data loading
- **SOTA Techniques**: RoPE, SwiGLU, RMSNorm, KV-cache, top-p/top-k sampling
## Quick Start
```bash
# Install dependencies
pip install -r requirements.txt
# Train + Generate (default)
python3 -m src.s00_main train+generate
# Train only
python3 -m src.s00_main train --epochs 20 --batch-size 4
# Generate from checkpoint
python3 -m src.s00_main generate --temperature 0.85
```
## Project Structure
```
music_gen_llm/
├── src/
│ ├── s00_main.py # Entry point — orchestrates pipeline
│ ├── s01_config.py # All configuration dataclasses
│ ├── s02_tokenizer.py # REMI MIDI tokenizer
│ ├── s03_dataset.py # Data download + tokenization + DataLoader
│ ├── s04_model.py # MusicTransformer (LLaMA-style)
│ ├── s05_trainer.py # Training loop with AMP + checkpointing
│ ├── s06_generator.py # Autoregressive generation with KV-cache
│ └── s07_utils.py # Logging, memory monitoring, seeding
├── tests/
│ └── test_pipeline.py # 7 unit tests covering all components
├── scripts/
│ ├── download_data.sh # Dataset setup
│ ├── train.sh # Training launcher
│ └── generate.sh # Generation launcher
├── docs/
│ ├── README.md # This file
│ ├── HLD.md # High-Level Design
│ ├── LLD.md # Low-Level Design
│ └── flow_diagram.drawio # Execution flow diagram
├── data/ # Downloaded MIDI + tokenized cache
├── checkpoints/ # Saved model weights
├── output/ # Generated MIDI files
├── requirements.txt
├── Dockerfile
└── .gitignore
```
## Execution Flow
```
s00_main.py → s01_config.py → s02_tokenizer.py → s03_dataset.py → s04_model.py → s05_trainer.py → s06_generator.py
│ │ │ │ │ │ │
Entry point Load configs Init tokenizer Download & tokenize Build model Train loop Generate MIDI
```
## Model Specifications
| Parameter | Value |
|---------------|-------------------------------------|
| Dim | 256 |
| Layers | 6 |
| Heads | 8 (Q) / 4 (KV) — GQA |
| Hidden (FFN) | 448 (SwiGLU) |
| Max Seq Len | 1024 |
| Vocab Size | 485 (REMI tokens) |
| Parameters | ~5M |
| Precision | BF16/FP16 (AMP) |
## Algorithms Used
1. **Rotary Position Embeddings (RoPE)** — Su et al. 2021
2. **Grouped Query Attention (GQA)** — Ainslie et al. 2023, from LLaMA-2
3. **SwiGLU Activation** — Shazeer 2020, from LLaMA
4. **RMS Layer Normalization** — Zhang & Sennrich 2019
5. **REMI Tokenization** — Huang & Yang 2020
6. **Cosine Annealing with Warmup** — Loshchilov & Hutter 2017
7. **Gradient Checkpointing** — Chen et al. 2016
8. **KV-Cache** — Standard for efficient autoregressive decoding
9. **Nucleus (Top-p) + Top-k Sampling** — Holtzman et al. 2020
10. **Repetition Penalty** — Keskar et al. 2019
|