README.md

---
language: en
tags:
  - audio
  - music
  - guitar
  - chart-generation
  - transformer
  - encoder-decoder
license: mit
---

# Tab Hero — ChartTransformer

An encoder-decoder transformer that generates guitar/bass charts from audio. Given a mel spectrogram, the model autoregressively produces a sequence of note tokens compatible with Clone Hero (`.mid` + `song.ini`).

## Model Description

| Property | Value |
|---|---|
| Architecture | Encoder-Decoder Transformer |
| Parameters | ~150M (Large config) |
| Audio input | Mel spectrogram (22050 Hz, 128 mels, hop=256) |
| Output | Note token sequence |
| Vocabulary size | 740 tokens |
| Training precision | bf16-mixed |
| Best validation loss | 0.1085 |
| Trained for | 65 epochs (195,030 steps) |

### Architecture

The **audio encoder** projects mel frames through a linear layer then a Conv1D stack with 4x temporal downsampling (~46ms per frame). The **decoder** is a causal transformer with:
- RoPE positional encoding (enables generation beyond training length)
- Flash Attention 2 via `scaled_dot_product_attention`
- SwiGLU feed-forward networks
- Difficulty and instrument conditioning embeddings
- Weight-tied input/output embeddings

Full architecture details: [docs/architecture.md](https://github.com/MattGroho/tab-hero/blob/main/docs/architecture.md)

### Tokenization

Each note is a 4-token quad: `[TIME_DELTA] [FRET_COMBINATION] [MODIFIER] [DURATION]`

| Range | Type | Count | Description |
|---|---|---|---|
| 0 | PAD | 1 | Padding |
| 1 | BOS | 1 | Beginning of sequence |
| 2 | EOS | 1 | End of sequence |
| 3–503 | TIME_DELTA | 501 | Time since previous note (10ms bins, 0–5000ms) |
| 504–630 | FRET | 127 | All non-empty subsets of 7 frets |
| 631–638 | MODIFIER | 8 | HOPO / TAP / Star Power combinations |
| 639–739 | DURATION | 101 | Sustain length (50ms bins, 0–5000ms) |

### Conditioning

The model supports 4 difficulty levels (Easy / Medium / Hard / Expert) and 4 instrument types (lead / bass / rhythm / keys), passed as integer IDs at inference time.

## Usage

```python
import torch
from tab_hero.model.chart_transformer import ChartTransformer
from tab_hero.data.tokenizer import ChartTokenizer

tok = ChartTokenizer()

model = ChartTransformer(
    vocab_size=tok.vocab_size,
    audio_input_dim=128,
    encoder_dim=768,
    decoder_dim=768,
    n_decoder_layers=8,
    n_heads=12,
    ffn_dim=3072,
    max_seq_len=8192,
    dropout=0.1,
    audio_downsample=4,
    use_flash=True,
    use_rope=True,
)

ckpt = torch.load("best_model.pt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model_state_dict"])
model.eval()

# audio_mel: (1, n_frames, 128) mel spectrogram tensor
tokens = model.generate(
    audio_embeddings=audio_mel,
    difficulty_id=torch.tensor([3]),   # 0=Easy 1=Medium 2=Hard 3=Expert
    instrument_id=torch.tensor([0]),   # 0=lead 1=bass 2=rhythm 3=keys
    temperature=1.0,
    top_k=50,
    top_p=0.95,
)
```

See [`notebooks/inference_demo.ipynb`](https://github.com/MattGroho/tab-hero/blob/main/notebooks/inference_demo.ipynb) for a full end-to-end example including audio loading and chart export.

## Training

- **Optimizer**: AdamW (lr=1e-4, weight_decay=0.01, betas=(0.9, 0.95))
- **Scheduler**: Cosine annealing with 1000-step linear warmup
- **Batch size**: 16 (effective 32 with gradient accumulation)
- **Gradient clipping**: max norm 1.0
- **Early stopping**: patience 15 epochs

## Limitations

- Trained on a specific dataset of Clone Hero charts; quality varies by genre and playing style.
- Source separation (HTDemucs) is recommended for mixed audio but not required.
- Mel spectrograms are lossy — the model cannot recover audio from its inputs.
- Output requires post-processing via `SongExporter` to produce playable chart files.

## Repository

[https://github.com/MattGroho/tab-hero](https://github.com/MattGroho/tab-hero)