FIGMA retrieves music from natural-language descriptions that specify fine-grained musical attributes — tempo, key, chord progression, beat/time signature — in addition to high-level attributes like genre and mood. This repository hosts the trained model checkpoint and the code to run inference.

Paper: https://arxiv.org/abs/2606.06615
Project page: https://nishitanand.github.io/figma-website
Code: https://github.com/nishitanand/FIGMA
Benchmark: https://huggingface.co/datasets/nishitanand/FGMCaps-benchmark
Data: https://huggingface.co/datasets/nishitanand/FGMCaps-data

Installation

bash

pip install torch transformers librosa soundfile numpy
pip3 install muq

Download the checkpoint

python

from huggingface_hub import snapshot_download
import os

local_dir = snapshot_download(repo_id="nishitanand/FIGMA")
ckpt = os.path.join(local_dir, "figma.ckpt")

Inference

python

import torch, librosa
from figma_model import Figma, get_tokenizer

device = "cuda"
model = Figma.from_checkpoint(ckpt, device=device)   # ckpt from the download step
tok = get_tokenizer()

# Text embedding
t = tok(["a song in F minor at 120 BPM in 4/4 time"], return_tensors="pt",
        padding="max_length", truncation=True, max_length=128).to(device)
text_emb = model.encode_text(t)              # [1, 512], L2-normalized

# Audio embedding (24 kHz mono -> [B, 1, samples])
wav, _ = librosa.load("clip.wav", sr=24000)
wav = torch.tensor(wav)[None, None].to(device)
audio_emb = model.encode_audio(wav)          # [1, 512], L2-normalized

similarity = (audio_emb @ text_emb.T).item()

Command-line text→audio retrieval over a folder:

bash

python inference.py --checkpoint figma.ckpt \
    --audio_dir ./clips --query "a jazzy track in C major at 90 bpm" --topk 5

Intended use & limitations

Intended for non-commercial research on music retrieval / music–text alignment.
English-centric captions; fine-grained attributes cover tempo, key, chords, and beat/time signature.

Citation

bibtex

@inproceedings{figma2026,
  title     = {FIGMA: Towards FIne-Grained Music retrievAl},
  author    = {Anand, Nishit and Seth, Ashish and Ghosh, Sreyan and Manocha, Dinesh and Duraiswami, Ramani},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.06615}
}

License

Released for non-commercial research use under CC BY-NC 4.0. The checkpoint bundles the MuQ encoder (weights CC BY-NC 4.0); the E5 encoder is MIT. You must attribute MuQ and E5 and comply with CC BY-NC 4.0. The inference code is MIT-licensed in the code repository.