ATOMM-1-toypoodle

ATOMM-1-toypoodle is an experimental autoregressive Transformer for symbolic music generation. The model is trained directly on MIDI files using a custom event-based tokenization that represents timing, instruments, note durations, pitches, and velocities.

This repository is intended as a research prototype rather than a production-ready music generation system.

Help

Inquiries/donations go to benjaminsnautilus@gmail.com. Would be nice if I got grants or a real GPU.

Model

The model consists of:

• Decoder-only Transformer architecture
• Rotary Position Embeddings (RoPE)
• PyTorch scaled dot-product attention
• KV-cache support for efficient autoregressive generation
• LayerNorm + GELU feed-forward blocks
• Mixed precision (torch.amp)
• Optional 8-bit Adam optimizer via bitsandbytes
• FlashAttention-compatible PyTorch attention kernels when supported by the hardware

The architecture size (embedding dimension, number of layers, and attention heads) is configurable at training time.

Tokenization

MIDI files are converted into sequences of discrete events.

The vocabulary contains:

• Start token
• Time-shift (gap) tokens
• Duration tokens
• Instrument/program tokens
• Pitch tokens
• Velocity tokens (32 bins)
• End token

Timing is quantized to 8-32 steps per beat.

Velocity values are quantized from MIDI's 128 values into 32 discrete bins.

Training

Using: a RTX 6000 Pro Blackwell, we are able to train models up to 1B efficiently and generalize under low budgets, with 30M version being 10¢ in compute from start to finish, with the 10B variant aiming under 5,000$ Training:

• Parses MIDI datasets in parallel
• Caches tokenized sequences
• Trains with next-token prediction (cross-entropy loss)
• Uses teacher forcing
• Supports checkpoint resume
• Saves checkpoints throughout training

Each training example is one fixed-length token sequence.

Generation

Generation is fully autoregressive.

Sampling supports:

• Temperature
• Top-k sampling
• Top-p (nucleus) sampling
• KV-cache decoding

Generated event sequences are converted back into standard MIDI files.

Currently, we support generation via our Google Colab page. Import model.pth and you're ready.

Requirements

Main dependencies include:

• PyTorch
• miditoolkit
• tqdm
• bitsandbytes

CUDA is recommended for training and generation.

Future Work

Future ATOMM-1 models will follow this naming progression, per version:

1. toy-poodle (38.7M parameters) (out)
2. little-green-ant (~150M parameters (out)
3. midi-copilot (~700M) (future)
4. chonky-production (~4B) (future)
5. planet-eater (~10B parameters) (future)

Together, these models are intended to cover a range from approximately 30 million to 10 billion parameters.

License

This project is released under the Apache License 2.0.

Citation

If you use this model or code in research, you may cite:

@misc{atomm1toypoodle,
  title={ATOMM-1-toypoodle},
  author={Benjamin Ecker},
  year={2026},
  publisher={Hugging Face}
}