ATOMM-1-little-green-ant is an experimental autoregressive Transformer for symbolic music generation. The model is trained directly on MIDI files using a custom event-based tokenization that represents timing, instruments, note durations, pitches, and velocities.

This repository is intended as a research prototype rather than a production-ready music generation system.

Help

Inquiries/donations go to benjaminsnautilus@gmail.com. Would be nice if I got grants or a real GPU.

Model

The model consists of:

Decoder-only Transformer architecture
Rotary Position Embeddings (RoPE)
PyTorch scaled dot-product attention
KV-cache support for efficient autoregressive generation
LayerNorm + GELU feed-forward blocks
Mixed precision (torch.amp)
Optional 8-bit Adam optimizer via bitsandbytes
FlashAttention-compatible PyTorch attention kernels when supported by the hardware

The architecture size (embedding dimension, number of layers, and attention heads) is configurable at training time.

Tokenization

MIDI files are converted into sequences of discrete events.

The vocabulary contains:

Start token
Bar token
Time-shift (gap) tokens
Duration tokens
Instrument/program tokens
Pitch tokens
Velocity tokens (32 bins)
End token

Timing is quantized to 8-32 steps per beat.

Velocity values are quantized from MIDI's 128 values into 32 discrete bins.

Training

Using: a RTX 6000 Pro Blackwell, we are able to train models up to 1B efficiently and generalize under low budgets, with 30M version being 10¢ in compute from start to finish, with the 10B variant aiming under 5,000$ Training:

Parses MIDI datasets in parallel
Caches tokenized sequences
Trains with next-token prediction (cross-entropy loss)
Uses teacher forcing
Supports checkpoint resume
Saves checkpoints throughout training

Each training example is one fixed-length token sequence.

Generation

Generation is fully autoregressive.

Sampling supports:

Temperature
Top-k sampling
Top-p (nucleus) sampling
KV-cache decoding

Generated event sequences are converted back into standard MIDI files.

Currently, we support generation via our Google Colab page. Import model.pth and you're ready.

Requirements

Main dependencies include:

PyTorch
miditoolkit
tqdm
bitsandbytes

CUDA is recommended for training and generation.

Future Work

Future ATOMM-1 models will follow this naming progression, per version:

toy-poodle (38.7M parameters) (out)
little-green-ant (~150M parameters (out)
midi-copilot (~700M) (training)
chonky-production (~4B) (future)
planet-eater (~10B parameters) (future)

Together, these models are intended to cover a range from approximately 30 million to 10 billion parameters.

License

This project is released under the Apache License 2.0.

Citation

If you use this model or code in research, you may cite:

bibtex

@misc{atomm1toypoodle,
  title={ATOMM-1-little-green-ant},
  author={Benjamin Ecker},
  year={2026},
  publisher={Hugging Face}
}

ResonML/ATOMM-1-little-green-ant

ATOMM-1-little-green-ant