Compose & Embellish: Piano Performance Generation Pipeline

• Shih-Lun Wu and Yi-Hsuan Yang
  "Compose & Embellish: Well-Structured Piano Performance Generation via A Two-Stage Approach."
  Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), 2023

Model characteristics

Stage 1: "Compose" model

• Model backbone: 12-layer Transformer w/ relative positional encoding
• Num trainable params: 41.3M
• Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
• Pretraining dataset: subset of Lakh MIDI full (LMD-full), 14934 songs
  • melody extraction (and data filtering) done by matching lyrics to tracks: https://github.com/gulnazaki/lyrics-melody/blob/main/pre-processing/create_dataset.py
  • structural segmentation done with A* search: https://github.com/Dsqvival/hierarchical-structure-analysis
• Finetuning dataset: subset of AILabs.tw Pop1K7 (Pop1K7), 1591 songs
  • melody extraction done with skyline algorithm: https://github.com/wazenmai/MIDI-BERT/blob/CP/melody_extraction/skyline/analyzer.py
  • structural segmentation done in the same way as pretraining dataset
• Training sequence length: 2400

Stage 2: "Embellish" model

• embellish_model_gpt2_pop1k7_loss0.398.bin
  • Model backbone: 12-layer GPT-2 Transformer (implementation)
  • Num trainable params: 38.2M
• embellish_model_pop1k7_loss0.399.bin (requires fast-transformers package, which is outdated as of Jul. 2024)
  • Model backbone: 12-layer Performer (paper, implementation)
  • Num trainable params: 38.2M
• Token vocabulary: Revamped MIDI-derived events (REMI) w/ slight modifications
• Training dataset: AILabs.tw Pop1K7 (Pop1K7), 1747 songs
• Training sequence length: 3072

BibTex

@inproceedings{wu2023compembellish,
  title={{Compose \& Embellish}: Well-Structured Piano Performance Generation via A Two-Stage Approach},
  author={Wu, Shih-Lun and Yang, Yi-Hsuan},
  booktitle={Proc. Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2023},
  url={https://arxiv.org/pdf/2209.08212.pdf}
}