CLaMP 2: Multimodal Music Information Retrieval Across 101 Languages Using Large Language Models

Overview

Links

• CLaMP 2 Code
• CLaMP 2 Paper
• CLaMP 2 Model Weights
• M3 Model Weights

Repository Structure

• code/: Includes scripts for training the CLaMP 2 and M3 models and extracting features from music and text data. You can modify hyperparameters and file paths in the configuration files to suit your training needs.
• music_classification/: Contains scripts for performing classification tasks via linear probe using the extracted features. This directory includes utilities for training linear classification models and making predictions on new feature data.
• process_data/: Provides tools to convert between various musical data formats (ABC notation, MusicXML, MIDI, and MTF) and to summarize music metadata with GPT-4. Before using CLaMP 2 or M3, you should use these scripts to convert your files into interleaved ABC notation or MTF compatible with these models.
• semantic_search/: Provides scripts for evaluating model performance, conducting semantic searches, and calculating similarity metrics based on extracted feature vectors.

Getting Started

Environment Setup

Data Preparation

1. Convert Files: Navigate to the process_data/ folder and convert your music files to a compatible format (interleaved ABC notation or MTF) suitable for CLaMP 2 and M3.
   • Use the conversion scripts in this folder for tasks like converting MusicXML to ABC and MIDI to MTF.
   • After collecting MusicXML (sheet music) or MIDI (performance data), perform the following operations to convert them into interleaved ABC notation or MTF respectively for model training:
     1. Obtain Interleaved ABC Notation:
        • Convert MusicXML files to ABC using batch_xml2abc.py.
        • Process the ABC files into interleaved notation using batch_interleaved_abc.py.
     2. Obtain MTF:
        • Convert MIDI files to MTF format using batch_midi2mtf.py.
     3. Convert Interleaved ABC Back to XML (Optional):
        • Use batch_xml2abc.py to convert interleaved ABC files back to MusicXML.
     4. Convert MTF Back to MIDI (Optional):
        • Use batch_mtf2midi.py to convert MTF files back to MIDI format.
2. Generate Multilingual Metadata Summaries: After converting the music files, the next step is to generate multilingual summaries of the music metadata. This is done using the gpt4_summarize.py script, which leverages the GPT-4 API to create structured summaries in both English and a randomly selected non-English language.
   Input Example: The input to the summarization script consists of a JSON file representing the music metadata. Here’s an example of a music entry in JSON format:

1. Merge all JSON files into a single JSONL file.
2. Place the merged JSONL file and the shortest common root directories (e.g., abc/ and/or mtf/) in the same folder, structured like this:

/your-target-folder/
├── abc/
├── mtf/
├── merged_output.jsonl

Training and Feature Extraction

1. Training Models: If you want to train CLaMP 2 or M3 models, check the scripts in the code/ folder.
   • Modify the config.py files to set your training hyperparameters and paths.
2. Extracting Features: After training, or if you have pre-trained models, you can extract features from your data using the respective scripts in the code/ folder.

Classification and Retrieval

1. Classification: If you need to classify the extracted features, navigate to the music_classification/ directory.
   • Here, you'll find scripts to train linear classification models and perform inference on new data.
2. Semantic Search: To perform semantic searches using the extracted features, refer to the scripts in the semantic_search/ folder.

Benchmarks

Citation