You can deploy this model and start earning money today!

m0yosore/
harmonia-multimodal-music-intelligence

Name: harmonia-multimodal-music-intelligence
Availability: InStock

Views

No views yet

Harmonia — Multimodal Music Intelligence Stack

Author: Moyosore Ogunjobi
What this is: A masters-level, end-to-end system for music tagging, emotion estimation, text-to-audio search, and recommendation, trained from multiple datasets and stitched together with a calibrated Mixture-of-Experts fusion model.

Why Harmonia exists

Music ML is usually siloed: tagging models ignore symbolic structure, recommenders ignore audio, and search systems ignore user behavior. Harmonia unifies these signals into a shared representation and exposes them through a single pipeline.

What it can do

Auto-tagging: genre / instrument / mood-theme (multi-label)
Emotion regression: valence + arousal (continuous)
Text ↔ audio retrieval: search audio by natural-language prompts
Recommendation: top-K ranking with LightGCN trained on playcount graphs
Fusion MoE: combines specialized experts into a calibrated final predictor

System overview (6 trainable models)

AudioTagger — CLAP/HTSAT audio encoder + multi-label head
AudioEmotion — shared audio encoder + valence/arousal head
ContrastiveAdapter — InfoNCE alignment (audio ↔ tag-captions)
SymbolicMusicBERT — MIDI-based understanding backbone + task head
RecLightGCN — collaborative filtering on user-song graphs
FusionMoE — mixture-of-experts + temperature calibration

Datasets

Place datasets under data/raw/:

MTG-Jamendo: data/raw/mtg_jamendo/
FMA: data/raw/fma/
DEAM: data/raw/deam/
Lakh MIDI: data/raw/lakh_midi/
MSD Taste Profile: data/raw/msd_taste_profile/

Run:

python scripts/check_data.py

Install

bash

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Train (each model separately)

bash

python -m harmonia.training.train_audio_tagger --config configs/audio_tagger_small.yaml
python -m harmonia.training.train_audio_emotion --config configs/audio_emotion_small.yaml
python -m harmonia.training.train_contrastive_adapter --config configs/contrastive_small.yaml
python -m harmonia.training.train_musicbert --config configs/musicbert_small.yaml
python -m harmonia.training.train_lightgcn --config configs/lightgcn_small.yaml
python -m harmonia.training.train_fusion_moe --config configs/fusion_small.yaml

Or run everything:

bash scripts/train_all.sh

Build the retrieval index (FAISS)

bash scripts/build_faiss.sh

Demo (Gradio)

python app.py

Results

This section is populated by saved metrics files from the training scripts:

Tagging: mAP / ROC-AUC
Emotion: RMSE / CCC
Retrieval: Recall@10 / MRR
Rec: Recall@20 / NDCG@20

How to use the pipeline (Python)

python

from harmonia.inference.pipeline import HarmoniaPipeline

pipe = HarmoniaPipeline.from_pretrained("./outputs/harmonia_bundle")
out = pipe.predict(audio_path="example.wav", text_query="sad jazzy piano at night")
print(out["tags"], out["emotion"], out["retrieval_topk"][:5])

License & data policy

This repo ships code + trained weights only. Datasets remain under their original licenses. Do not redistribute copyrighted audio, MIDI, or lyrics.

m0yosore/harmonia-multimodal-music-intelligence