File size: 3,943 Bytes
30e9297 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | # High-Level Design (HLD)
## System Overview
Music Generation LLM is a symbolic music generation system that learns patterns from
classical MIDI music and generates new compositions. It operates entirely on symbolic
note representations (MIDI events), not raw audio waveforms.
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────┐
│ USER INTERFACE │
│ CLI: train / generate / both │
└────────────────────────┬────────────────────────────────┘
│
┌────────────────────────▼────────────────────────────────┐
│ ORCHESTRATOR (s00_main) │
│ Parses args, wires components │
└──────┬─────────────────┬────────────────────┬───────────┘
│ │ │
┌──────▼──────┐ ┌───────▼───────┐ ┌────────▼────────┐
│ DATA │ │ MODEL │ │ GENERATION │
│ PIPELINE │ │ PIPELINE │ │ PIPELINE │
├─────────────┤ ├───────────────┤ ├─────────────────┤
│ HF Download │ │ MusicTrans- │ │ Autoregressive │
│ MIDI Parse │ │ former Build │ │ w/ KV-cache │
│ REMI Token │ │ Train Loop │ │ Top-p/k Sample │
│ DataLoader │ │ AMP + Ckpt │ │ MIDI Export │
└─────────────┘ └───────────────┘ └─────────────────┘
```
## Component Responsibilities
### Data Pipeline (s02 + s03)
- Download 4,796 MIDI files from HuggingFace
- Parse with `pretty_midi` library
- Tokenize using REMI scheme (Note On/Off, Velocity, TimeShift, Bar, Position)
- Create PyTorch DataLoaders with padding and random cropping
### Model (s04)
- LLaMA-style Transformer (6 layers, 256 dim, 8 heads)
- Grouped Query Attention with 4 KV heads (50% memory reduction)
- SwiGLU feed-forward with RMSNorm
- RoPE for positional encoding
- Gradient checkpointing support
### Training (s05)
- AdamW optimizer with cosine LR warmup
- Mixed precision (BF16/FP16) via PyTorch AMP
- Gradient accumulation (effective batch = 32)
- Early stopping with patience
- TensorBoard logging
### Generation (s06)
- Autoregressive decoding with KV-cache
- Temperature scaling + Top-k + Top-p (nucleus) sampling
- Repetition penalty for diverse output
- Direct MIDI file export
## Data Flow
```
MIDI Files → REMI Tokens → Training → Trained Model → Generation → MIDI Output
```
## Constraints & Decisions
| Decision | Rationale |
|------------------------------|----------------------------------------------|
| Symbolic (MIDI) not audio | 100x smaller data, trainable on consumer GPU |
| GQA over standard MHA | 50% KV-cache memory reduction |
| Gradient checkpointing | ~50% memory savings, ~20% speed cost |
| BF16 mixed precision | 50% memory, maintains numeric stability |
| REMI tokenization | Best published results for symbolic music |
| 5M params (not 100M+) | Fits in <=4GB VRAM, fast training |
|