MOSS-Music is an open-source music understanding model from
MOSI.AI, the OpenMOSS team,
and Shanghai Innovation Institute. Built on the same
audio backbone as MOSS-Audio,
MOSS-Music is further specialised on music via dedicated continual pre-training
and supervised fine-tuning — targeting musical captioning, lyrics ASR,
structural analysis, chord / key / tempo reasoning, and long-form musical
question answering. In this release, we provide two 8B models:
MOSS-Music-8B-Instruct and MOSS-Music-8B-Thinking. The Instruct variant
is optimised for direct instruction following on musical prompts, while the
Thinking variant provides stronger chain-of-thought reasoning for musical
analysis.
Music is not just audio plus lyrics — understanding it requires perceiving
harmonic structure, rhythm, timbre, instrumentation, performance nuance, and
the textual content of the lyrics, and reasoning about them jointly across
time. MOSS-Music is built to unify these capabilities within a single
model.
Lyrics ASR & time-aligned transcription: Accurate singing ASR with
sentence- and word-level timestamps, robust to backing tracks.
Musical captioning & tagging: Natural-language descriptions of mood,
genre, instrumentation, production style, and emotional trajectory.
Key / tempo / chord reasoning: Identifies musical key, beats, downbeats,
and chord progressions, including timestamped chord transcription.
Structural analysis: Segments a song into intro / verse / chorus /
bridge / outro and reasons about repetition and contrast.
Musical QA and long-form analysis: Open-ended question answering
grounded in a full track, including chain-of-thought reasoning in the
Thinking variant.
MOSS-Music overview
Model Architecture
MOSS-Music inherits the MOSS-Audio modular design, comprising three
components: an audio encoder, a modality adapter, and a large language model.
Raw audio is first encoded by MOSS-Audio-Encoder into continuous temporal
representations at 12.5 Hz, which are then projected into the language
model's embedding space through the adapter and finally consumed by the LLM
for auto-regressive text generation.
Rather than relying on off-the-shelf audio frontends, we train a dedicated
encoder from scratch to obtain more robust acoustic representations, tighter
temporal alignment, and better extensibility across musical styles, singing,
and non-speech acoustic content.
DeepStack Cross-Layer Feature Injection
Using only the encoder's top-layer features tends to lose low-level prosody,
transient events, and local time-frequency structure. To address this, we
adopt a DeepStack-inspired cross-layer injection module between the
encoder and the language model: in addition to the encoder's final-layer
output, features from earlier and intermediate layers are selected,
independently projected, and injected into the language model's early layers,
preserving multi-granularity information from low-level acoustic details to
high-level semantic abstractions.
This design is especially well-suited for music understanding, as it helps
retain rhythm, timbre, transients, and instrumental texture — information
that a single high-level representation cannot fully capture, yet is critical
for chord recognition, structural analysis, and nuanced musical description.
Time-Aware Representation
Time is a critical dimension in music understanding. To enhance explicit
temporal awareness, we adopt a time-marker insertion strategy during
pre-training: explicit time tokens are inserted between audio frame
representations at fixed time intervals to indicate temporal positions.
This design enables the model to learn "what happened when" within a unified
text generation framework, naturally supporting timestamped lyrics ASR,
beat / downbeat localisation, section boundary detection, and long-song
retrospective QA.
Building on the MOSS-Audio backbone, MOSS-Music is further enhanced through:
continual pre-training on a large, diverse music corpus produced by
the data annotation and processing pipeline
MOSS-Music-Data-Pipeline,
with an emphasis on singing, lyrics, and full-song coverage;
supervised fine-tuning (SFT) on music-centric instruction data covering
captioning, lyrics ASR, chord / key / structural analysis, and long-form
musical QA;
additional reasoning tuning for the Thinking variant.
Smaller (4B) variants and additional sizes may follow. Stay tuned!
Music Data Pipeline
The training data used by MOSS-Music is produced by an end-to-end pipeline
that goes from raw audio to chat-formatted training samples. That pipeline is
available at
MOSS-Music-Data-Pipeline,
which hosts duration detection, MIR feature extraction, song-structure
segmentation, lyrics ASR, metadata cleanup, and ALM-driven caption / query
generation with models such as Qwen3-Omni, MusicFlamingo, and other
audio-language models.
Evaluation
We evaluate MOSS-Music on a diverse suite of public music understanding
benchmarks. Key results:
Music QA and understanding: MOSS-Music-8B-Instruct achieves 80.38
average accuracy across 8 public music QA benchmarks (excluding the
three NSynth note-recognition tracks), ranking first among all compared
models in our current evaluation set.
Music captioning: In our preliminary GPT-5.4-as-a-Judge evaluation,
the MOSS-Music series leads both caption benchmarks, with
MOSS-Music-8B-Thinking reaching 4.53 on MusicCaps and
MOSS-Music-8B-Instruct reaching 4.58 on SDD.
Lyrics ASR for singing voice: MOSS-Music-8B-Thinking achieves the
best average lyrics recognition error across MUSDB18, MIR-1K and
Opencpop (15.88% avg WER/CER), clearly ahead of all compared
audio-language baselines including Gemini-3.1-Pro-Preview,
MusicFlamingo and Qwen3-Omni. Detailed timestamped-ASR results will be
released in a later update.
Chord transcription: MOSS-Music supports chord transcription, including
timestamped chord transcription for harmonic analysis, accompaniment
reference, and related downstream use cases. Detailed benchmark results will
be released in a later update.
Music QA & Understanding (Accuracy↑)
Model
MMAU-music
MMAU-mini-music
MMAU-Pro-music
MMAR-music
MuChoMusic
Music-AVQA
NSynth (instrument)
NSynth (source)
NSynth (pitch)
GTZAN
Medley-Solos-DB
Avg
MOSS‑Music‑8B‑Instruct
79.33
80.78
71.02
59.70
89.39
76.78
86.55
61.07
86.94
93.59
92.42
80.38
Gemini‑3.1‑Pro
71.69
77.18
73.06
71.64
79.53
61.51
13.38
38.90
6.47
86.39
80.34
75.17
MOSS‑Music‑8B‑Thinking
74.09
77.78
67.98
50.25
82.90
68.90
56.17
57.48
77.83
84.78
87.42
74.26
MusicFlamingo
76.83
76.35
65.60
48.66
74.58
73.60
80.76
75.89
0.00
84.45
90.86
73.87
Audio‑Flamingo‑Next
72.39
72.07
61.64
45.27
75.62
62.94
86.40
66.73
0.05
77.68
91.47
69.89
MiMo‑Audio‑7B‑Instruct
66.36
72.97
66.50
45.77
75.40
57.05
25.01
1.49
4.86
65.67
93.81
67.94
Step‑Audio‑R1
66.46
75.08
62.34
50.75
72.62
57.98
13.75
15.87
2.39
73.67
82.45
67.67
Qwen3‑Omni
65.76
68.77
66.27
48.54
78.77
56.05
30.92
44.30
28.08
80.15
69.65
66.75
Kimi‑Audio‑7B‑Instruct
47.95
52.25
59.10
45.27
70.18
68.90
6.01
0.81
3.88
39.54
71.98
56.90
Avg is computed over 8 public music QA benchmarks:
MMAU-music, MMAU-mini-music, MMAU-Pro-music, MMAR-music,
MuChoMusic, Music-AVQA, GTZAN, and Medley-Solos-DB.
We exclude the three NSynth tracks from the main average because they focus
on fine-grained isolated-note recognition, including instrument-family,
acoustic/electronic source, and exact pitch discrimination from short
single-note clips. Some compared audio-language models are not explicitly
designed for this note-level classification setting, so we report NSynth
separately for reference rather than mixing it into the headline average.
Music Captioning
We further report a preliminary GPT-5.4-as-a-Judge music captioning
comparison on MusicCaps and Song Describer Dataset (SDD). Scores are on a
1-5 scale across 9 dimensions: genre/style, mood/affect, tempo/rhythm,
instrumentation/timbre, vocals, melody/harmony, structure/form,
production/audio quality, and scene/use case.
Overall caption quality: the MOSS-Music series remains strongest across
both caption benchmarks, with MOSS-Music-8B-Thinking reaching 4.53 on
MusicCaps and MOSS-Music-8B-Instruct reaching 4.58 on SDD.
Stronger structural descriptions: MOSS-Music shows the clearest gains on
structure / form / progression, especially on SDD.
Competitive baselines on instrumentation and scene semantics:
MusicFlamingo and Gemini-3.1-Pro remain competitive on
instrumentation/timbre, while Gemini-3.1-Pro is strongest on
scene / use case.
MusicCaps
Model
Genre
Mood
Tempo
Instr.
Vocals
Melody/Harmony
Structure
Production
Scene
Avg
MOSS‑Music‑8B‑Thinking
4.78
4.69
4.62
4.40
4.46
4.40
4.86
4.35
4.18
4.53
Gemini‑3.1‑Pro
4.70
4.60
4.48
4.68
4.18
4.18
3.86
4.40
4.72
4.42
MOSS‑Music‑8B‑Instruct
4.60
4.52
4.46
4.02
4.30
4.38
4.78
4.20
3.96
4.36
MusicFlamingo
4.80
4.36
4.50
4.64
3.94
4.08
3.58
4.30
3.72
4.21
Audio‑Flamingo‑Next
4.34
4.56
4.08
4.30
4.18
3.78
3.66
4.04
3.92
4.10
MiMo‑Audio‑7B‑Instruct
4.02
4.20
4.46
4.28
4.36
3.62
3.30
4.08
3.50
3.98
Step‑Audio‑R1
4.22
4.02
4.20
3.96
3.84
4.02
3.24
4.10
3.54
3.90
Qwen3‑Omni
4.58
4.50
4.26
3.62
3.64
3.48
2.98
4.18
4.42
3.96
Kimi‑Audio‑7B‑Instruct
3.98
3.92
4.32
3.88
4.48
3.28
2.72
3.72
3.24
3.73
Song Describer Dataset (SDD)
Model
Genre
Mood
Tempo
Instr.
Vocals
Melody/Harmony
Structure
Production
Scene
Avg
MOSS‑Music‑8B‑Instruct
4.84
4.76
4.68
4.24
4.52
4.56
4.92
4.42
4.24
4.58
Gemini‑3.1‑Pro
4.72
4.64
4.52
4.72
4.22
4.24
3.94
4.46
4.82
4.48
MOSS‑Music‑8B‑Thinking
4.66
4.58
4.50
4.36
4.36
4.44
4.84
4.26
4.02
4.45
MusicFlamingo
4.82
4.40
4.52
4.70
3.98
4.14
3.66
4.36
3.80
4.26
Audio‑Flamingo‑Next
4.40
4.62
4.14
4.36
4.22
3.84
3.74
4.10
4.00
4.16
MiMo‑Audio‑7B‑Instruct
4.08
4.26
4.52
4.34
4.42
3.70
3.38
4.16
3.58
4.05
Step‑Audio‑R1
4.30
4.10
4.26
4.02
3.92
4.10
3.32
4.18
3.62
3.98
Qwen3‑Omni
4.62
4.54
4.30
3.68
3.70
3.56
3.06
4.24
4.50
4.02
Kimi‑Audio‑7B‑Instruct
4.04
3.98
4.38
3.96
4.54
3.36
2.80
3.80
3.32
3.80
Lyrics ASR (WER / CER↓)
We further evaluate MOSS-Music on singing-voice lyrics ASR across three
representative benchmarks:
MUSDB18 — English pop songs with backing tracks, scored with WER;
MIR-1K — Chinese karaoke clips with background music, scored with CER;
Opencpop — clean Mandarin studio singing, scored with CER.
Avg is the unweighted mean of the three dataset-level error rates.
Model
MUSDB18 WER
MIR-1K CER
Opencpop CER
Avg
MOSS‑Music‑8B‑Thinking
29.19%
15.84%
2.60%
15.88%
MOSS‑Music‑8B‑Instruct
32.99%
23.96%
4.62%
20.52%
Gemini‑3.1‑Pro‑Preview
26.25%
36.37%
6.00%
22.87%
MusicFlamingo
23.41%
38.98%
18.73%
27.04%
Qwen3‑Omni‑30B‑A3B‑Instruct
62.67%
20.48%
2.26%
28.47%
MiMo‑Audio‑7B‑Instruct
94.16%
23.34%
6.77%
41.42%
Kimi‑Audio‑7B‑Instruct
97.53%
25.83%
4.90%
42.75%
Step‑Audio‑R1
81.67%
48.03%
4.15%
44.62%
Audio‑Flamingo‑Next
94.93%
55.63%
12.47%
54.34%
MOSS-Music-8B-Thinking achieves the lowest average lyrics-ASR error
(15.88%) across these three datasets, with particular gains on
MIR-1K (Chinese karaoke with accompaniment) and Opencpop (clean Mandarin
singing). MOSS-Music also inherits the strong timestamp-aware ASR ability
from MOSS-Audio; detailed singing-timestamp ASR results will be added soon.
Chord Transcription
MOSS-Music supports chord transcription, including timestamped chord
transcription that tracks chord progression over time. This can be useful for
harmonic analysis, accompaniment reference, music education, and related use
cases. Detailed benchmark results will be added soon.
Quickstart
Environment Setup
We recommend Python 3.12 with a clean Conda environment. The commands below
are enough for local inference.
[!IMPORTANT]
To achieve the best generation quality and fully leverage the model's capabilities, we
strongly recommend using SGLang Serving for inference.
See the full SGLang guide in moss_music_usage_guide.md.
You can replace ./weights/MOSS-Music-8B-Instruct with
./weights/MOSS-Music-8B-Thinking if needed.
If you use the default torch==2.9.1+cu128 runtime, installing
nvidia-cudnn-cu12==9.16.0.29 is recommended before starting sglang serve.
Gradio App
Start the Gradio demo with:
python app.py
The server address and port can be overridden via the
MOSS_MUSIC_SERVER_NAME and MOSS_MUSIC_SERVER_PORT environment variables,
and the default model ID via MOSS_MUSIC_MODEL_ID.
