Vocal Music Whisper
Fine-tunes of
laion/music-whisper
specialised on
vocals-only material — a cappella singing that sometimes carries a
backing choir and usually does not.
The base model is a general music captioner: it describes instrumentation, tempo, genre and
mood on full-band produced tracks. Confronted with dry unaccompanied voice it tends to
hallucinate a backing band that is not there. These checkpoints fix that and push the
description toward what actually matters in an a cappella clip: how many voices, vocal
technique, register, timbre and recording character.
✅ Default checkpoint: v2/unbalanced-lr3e5 — and it is at the repo root
1from transformers import WhisperForConditionalGeneration, WhisperProcessor
2# no subfolder= : the repo root IS the default checkpoint
3m = WhisperForConditionalGeneration.from_pretrained("TTS-AGI/vocal-music-whisper")
4p = WhisperProcessor.from_pretrained("TTS-AGI/vocal-music-whisper")
The same weights are also kept at v2/unbalanced-lr3e5/ so the ablation stays checkable. Every
other folder in this repo is an ablation or a superseded release; if you are not
deliberately comparing checkpoints, use the root.
Chosen on caption quality, judged by Gemini 3.6 Flash
given the audio and the
caption over all 158 held-out clips, twice
(
full method and raw judgements):
overall impression
3.47 / 5
against 3.29 for the balanced retrain,
3.23 for v1 and
2.48 for the base model.
The margin over the balanced retrain is inside the noise (+0.18,
95 % CI [-0.04, +0.39],
p = 0.09) — on caption quality alone these two are a
tie, and this card does not claim otherwise. The tie is broken by the axes that do
separate: v2/unbalanced-lr3e5 has the better validation loss
(0.9597 vs 0.9714),
the better voice-gender balanced accuracy
(61.6 % vs
60.6 %), and fewer factual errors flagged per
caption (1.41 vs
1.71). It is behind the balanced run on nothing
that was measured.
Fine-tuning did help captioning, which was not a foregone conclusion given the base
model wins the gender axis outright: +0.99 overall over
laion/music-whisper (CI [+0.69,
+1.28], p < 1e-4). The gender regression below is real
and is not cancelled by this — they are different axes and they disagree.
⚠️ Voice gender: still broken, and rebalancing the audio did not fix it
Every fine-tune here calls most female voices male. The base model does not. This was
first reported on ten clips; it has now been measured on the whole 158-clip held-out
split (34 female, 124 male by the Gemini audio judge), and a gender-balanced corpus was
built and trained on to try to fix it. The fix did not work, and the controls say why it
looked like it might.
Balanced accuracy (mean of the two recalls) is the honest summary. The split is
78 % male, so plain accuracy rewards a model for calling
everything male: the previously published frozen-encoder scores
79.7 % plain accuracy while getting
31 of the 34 female clips wrong.
| checkpoint | val loss | female recall | male recall | balanced acc. | abstains | mean words |
|---|
base — laion/music-whisper | — | 72.7 % | 73.4 % | 73.1 % | 0.6 % | 135 |
frozen-encoder (previous release, unbalanced) | 0.9461 | 8.8 % | 99.2 % | 54.0 % | 0.0 % | 137 |
full (previous release, unbalanced) | 1.1080 | 8.8 % | 100.0 % | 54.4 % | 0.0 % | 140 |
| unbalanced control, lr 3e-5 ← shipped | 0.9597 | 27.3 % | 96.0 % | 61.6 % | 0.6 % | 140 |
| unbalanced control, lr 1e-4 | 0.9348 | 20.6 % | 96.0 % | 58.3 % | 0.0 % | 139 |
| balanced, lr 3e-5 | 0.9968 | 11.8 % | 99.2 % | 55.5 % | 0.0 % | 141 |
| balanced, lr 1e-4 | 0.9714 | 23.5 % | 97.6 % | 60.6 % | 0.0 % | 137 |
Read the controls before crediting anything. orig_* are trained on the original,
unbalanced corpus on the same 10-epoch schedule as the balanced runs. The best balanced
accuracy of any fine-tune is the unbalanced control orig_lr3e5 at
61.6 %, ahead of both balanced runs
(60.6 % and 55.5 %). At a
matched learning rate the balanced run is worse than its control
(55.5 % vs 61.6 %). So the
movement from the previous release's 54.0 % comes from the
longer schedule with best-epoch selection, not from rebalancing the data.
Why rebalancing failed — measured, not guessed. The corpus was balanced on the audio
judge's label, but the training target is the caption text, and the two are not the same
thing. On the balanced set the audio is exactly
50.0 % female, yet 2,342 of its
3,189 female-voiced clips
(73 %) carry a caption that calls the singer
male. Balancing therefore moved the caption-text female share only from
5.6 % to 15.4 % —
2.8× — and female recall moved
8.8 % → 23.5 %, a factor of
2.7. The model
tracked the label it was actually trained on, not the one the corpus was balanced on.
The next attempt has to fix the captions, not the audio mix.
Until then, prefer laion/music-whisper if
voice gender matters to you — it is right 73.1 % of the
time on this split, better than any fine-tune here.
Gender is read out of the caption by a conservative token-majority rule; captions that
never commit count as abstains and are excluded from the recalls, so a model cannot score well
by saying less. The rule counts bass as a male token, which could favour models that talk
about instruments; recomputing with bass removed leaves the ranking unchanged. The
reference label is the Gemini audio judge — an instrument, not ground truth.
v2 checkpoints (this update)
Four more frozen-encoder runs, 10 epochs each with best-epoch selection, on the unchanged
158-clip validation split. They exist to answer one question -- does balancing the corpus by
voice gender fix the gender regression? -- and the answer is no. They are published because a
negative result that cost 1.25 GPU-h is
still a result, and because two of them are simply better than the v1 release.
| folder | corpus | lr | val loss | balanced acc. | why it is here |
|---|
v2/unbalanced-lr3e5 = repo root | original | 3e-5 | 0.9597 | 61.6 % | the default -- best caption quality and best gender of any fine-tune |
v2/unbalanced-lr1e4 | original | 1e-4 | 0.9348 | 58.3 % | best validation loss of any checkpoint here |
v2/balanced-lr1e4 | 50/50 balanced | 1e-4 | 0.9714 | 60.6 % | the gender-balanced retrain |
v2/ablation/balanced-lr3e5 | 50/50 balanced | 3e-5 | 0.9968 | 55.5 % | metadata only -- matched-lr control for unbalanced-lr3e5 |
Both v2/unbalanced-* beat the v1 frozen-encoder
(val 0.9461, balanced acc.
54.0 %) on both axes, so if you are using v1 today, switch. The v1
folders stay published so the comparison remains checkable.
1from transformers import WhisperForConditionalGeneration, WhisperProcessor
2r = "TTS-AGI/vocal-music-whisper"
3m = WhisperForConditionalGeneration.from_pretrained(r) # root = the default
4p = WhisperProcessor.from_pretrained(r)
5# identical weights, if you prefer to be explicit:
6# WhisperForConditionalGeneration.from_pretrained(r, subfolder="v2/unbalanced-lr3e5")
Every number above is regenerated from eval/ in this repo by
caption_vs_judge.py, eval_gender_val.py and make_card_v2.py; none is typed by hand.
Caption quality — how the default was chosen
The previous round of this card picked a checkpoint on voice-gender recall — one
attribute. The model's job is to write a good caption, and a checkpoint can win on gender
agreement while writing worse captions, so caption quality was measured separately and it is
what selects the default.
Method. Every checkpoint captioned all 158 held-out clips (the unchanged validation
split; greedy decoding, 440 new tokens — the most Whisper's 448-position decoder allows once
its three forced start tokens are counted). Gemini 3.6 Flash was then given the audio and
one caption and asked to score that caption on three axes, every level anchored in words
rather than left as a bare 0–5 scale. The judge is blind: it never learns which checkpoint
wrote the caption, never sees a second caption, and never sees the reference. It must write a
one-line "what I hear" and list the specific statements it believes are wrong before
scoring, so a surprising number can be traced to the perception behind it.
| axis | question |
|---|
| detail 0–5 | how specific and informative, versus vague boilerplate. Wrong facts are not penalised here, and length alone earns nothing. |
| factual correctness 0–5 | does it match what is actually audible — voice type, solo vs layered, instruments present or absent, genre, mood. This is why the audio is supplied. |
| overall impression 0–5 | the caption as a whole. Not an average; usefulness weighed against trustworthiness. |
Every clip was judged
twice independently (1,264 judgements in total,
0 failures, every response asserted to carry non-zero
AUDIO prompt tokens) and the two
passes are averaged, which halves the judge's own noise. Raw judgements:
eval/capjudge.jsonl; summary:
eval/capjudge_summary.json.
| checkpoint | detail | factual | overall | s.e.m. | errors flagged / caption | captions with none |
|---|
base — laion/music-whisper | 3.62 ± 0.50 | 2.57 ± 1.48 | 2.48 ± 1.38 | 0.11 | 2.42 | 15 % |
v1 frozen-encoder | 4.87 ± 0.29 | 3.21 ± 1.35 | 3.23 ± 1.35 | 0.11 | 1.65 | 23 % |
| v2 balanced lr 1e-4 | 4.86 ± 0.28 | 3.27 ± 1.45 | 3.29 ± 1.44 | 0.11 | 1.71 | 29 % |
| v2 unbalanced lr 3e-5 ← default | 4.91 ± 0.25 | 3.45 ± 1.43 | 3.47 ± 1.41 | 0.11 | 1.41 | 34 % |
n = 158 clips per checkpoint, each judged twice. Spread is the standard deviation across
clips — it is large because clips differ enormously in how hard they are to caption, which
is exactly why the comparisons below are paired.
The two v2 contenders are within noise of each other
| comparison | overall Δ | 95 % CI (bootstrap, paired) | p | clips won / tied / lost | verdict |
|---|
v2/unbalanced-lr3e5 − v2/balanced-lr1e4 | +0.18 | [-0.04, +0.39] | 0.09 | 64 / 43 / 51 | tie — do not read a winner into this |
v2/unbalanced-lr3e5 − v1 frozen-encoder | +0.24 | [+0.02, +0.47] | 0.04 | 63 / 39 / 56 | separated |
v2/unbalanced-lr3e5 − base | +0.99 | [+0.69, +1.28] | <1e-4 | 96 / 18 / 44 | separated |
Three things this evaluation says that the gender evaluation could not:
- Fine-tuning helps captioning, clearly. Every fine-tune beats the base model by roughly
three-quarters of a point of overall impression. That is worth stating plainly because the
base model beats every fine-tune on voice gender — the two axes genuinely disagree, and
only measuring both makes that visible.
- The two v2 contenders cannot be separated on caption quality. The gap between them
(+0.18) is smaller than the judge's own test–retest disagreement
(mean absolute difference 0.56 points on overall,
63 % exact agreement between the two passes).
The default goes to
v2/unbalanced-lr3e5 on the tie-breakers listed at the top of this card, not
on this number.
- Detail is at ceiling and factual correctness is the whole story. All three fine-tunes
score 4.86–4.91
on detail with 135 of 158 clips at the maximum: they
have all learned to write richly specific prose, and detail no longer discriminates between
them. What separates them from the base model, and what would separate a future checkpoint
from these, is whether the specifics are true.
Read this as a relative measurement, not an absolute grade. The judge is one model
listening to synthetic singing; on some clips it hears an instrument the reference caption
says is not there. That bias applies identically to all four checkpoints on identical audio,
which is what makes the paired differences meaningful and the absolute values only
indicative.
Voice gender is unchanged by the longer generation. Regenerating the captions at 440
tokens instead of 200 moves the balanced accuracy of all four checkpoints by
0.0 percentage points, so the gender table below stands as published.
🎧 Five held-out clips, captioned by the default checkpoint
Each clip was held out of training. Press play, then read what v2/unbalanced-lr3e5 says about it
and what the judge scored that exact caption. The last one is a failure, included
because five flattering clips would make a prettier card and a useless one.
g60_bedroom_pop_030 — bedroom pop
v2/unbalanced-lr3e5 says: This recording features a solo female voice performing an intimate indie-pop melody without any instrumental accompaniment. The singer employs a breathy, delicate delivery with a light mixed voice that occasionally breaks into a delicate falsetto for higher melodic peaks. Her timbre is warm and resonant yet possesses a fragile, airy quality that enhances the song's melancholic and nostalgic mood. The vocal range is primarily in the mezzo-soprano register, moving fluidly between a grounded chest voice and a light, airy head voice. The phrasing is predominantly legato, with subtle glottal onsets and a narrow, controlled vibrato that adds a sense of vulnerability to the performance. The recording is dry and close-miked, capturing the fine details of the vocal texture, with a light touch of reverb that provides a sense of space without masking the detail. The overall expressive arc is one of quiet reflection and longing, maintaining a consistent, intimate character throughout the clip.
Judge: detail 5.0, factual 5.0, overall 5.0 ·
what it heard: A solo female voice sings a gentle indie-pop song softly without any instrumental accompaniment.
· An excellent, highly descriptive, and accurate caption that perfectly fits the audio clip.
choir_1aad9c13bb_s5054 — choir
v2/unbalanced-lr3e5 says: A solo male baritone performs an intimate, a cappella folk melody with a warm and resonant timbre. The vocal delivery is characterized by a breathy, mixed-voice technique and a gentle, controlled vibrato that adds a sense of vulnerability. The phrasing is predominantly legato, with subtle glottal attacks and a slight breathiness that enhances the emotional intimacy of the performance. The recording is dry and close-miked, capturing the fine details of the singer's breath and articulation without any instrumental accompaniment or digital effects. The overall mood is one of quiet reflection and longing, maintained through a consistent expressive arc.
Judge: detail 5.0, factual 5.0, overall 5.0 ·
what it heard: A solo male baritone sings an a cappella folk-style song in a warm, intimate tone with no instrumental accompaniment.
· The caption is highly detailed, entirely accurate, and paints a precise picture of the audio recording.
g60_americana_singer_songwriter_011 — americana singer songwriter
v2/unbalanced-lr3e5 says: This recording features a solo male baritone performing a dramatic musical theater piece with a powerful, resonant delivery. The singer employs a strong mixed voice and belting technique, particularly in the higher register where the tone becomes more intense and slightly gritty. His phrasing is primarily legato, punctuated by clear glottal attacks and a controlled, medium-width vibrato that adds emotional weight to the sustained notes. The timbre is warm and full-bodied, conveying a sense of yearning and determination that builds throughout the clip. The recording is entirely unaccompanied, with no instrumental backing or background noise, and the sound is dry and close-miked with a subtle touch of reverb that enhances the natural resonance of the voice.
Judge: detail 5.0, factual 4.0, overall 4.0 ·
what it heard: A male baritone sings a theatrical piece solo and unaccompanied.
· The caption is exceptionally detailed and completely accurate, offering a precise description of the audio clip.
g60_metal_symphonic_soprano_019 — metal symphonic soprano
v2/unbalanced-lr3e5 says: This recording features a solo male voice that begins with a gentle, narrative quality, eventually joined by layered backing vocals and lush harmonies that create a rich, choral texture. The performance sits firmly within the contemporary folk or musical theater genre, characterized by its storytelling approach and clear, articulate diction. The singer employs a smooth legato phrasing with a controlled, medium-width vibrato that adds warmth to the sustained notes. The vocal timbre is resonant and warm, possessing a pure, pure quality that remains consistent across a baritone-tenor range. The expressive arc moves from a soothing, lullaby-like opening to a more rhythmic and harmonically dense conclusion. This is an entirely unaccompanied vocal performance with no audible instrumentation or background noise. The recording is polished and dry, with a subtle room reverb that provides a sense of space while maintaining a close, intimate feel.
Judge: detail 5.0, factual 3.0, overall 3.0 ·
what it heard: I hear an unaccompanied solo female voice singing a lullaby-like folk song. · flagged: solo male voice; baritone-tenor range
· While highly detailed and descriptive of the musical texture, the incorrect gender identification misleads the listener on a core element.
g60_melodie_francaise_044 — melodie francaise
v2/unbalanced-lr3e5 says: This recording features a solo male voice that begins with a conversational, intimate delivery before being joined by layered backing vocals and lush harmonies that create a powerful, anthemic sound. The genre is firmly rooted in Indie Folk, characterized by a warm, resonant timbre and a slight grit that adds emotional weight to the performance. The singer demonstrates a versatile range, moving from a breathy lower register to a soaring, belted mixed voice in the higher passages. The vocal technique includes subtle glottal attacks and a controlled vibrato that widens during sustained notes. The expressive arc is one of building intensity, starting with a quiet, nostalgic reflection and culminating in a passionate, yearning climax. The recording is entirely unaccompanied, featuring only the human voice with no instrumental backing or background noise. The production is polished and dry, with a close-miked quality that is enhanced by a touch of studio reverb, providing a sense of space without masking the detail of the vocal performance.
Judge: detail 4.0, factual 1.0, overall 1.0 ·
what it heard: I hear a solo female voice singing an pop/indie-country style phrase without any instrumental accompaniment or backing vocals. · flagged: solo male voice; joined by layered backing vocals and lush harmonies; soaring, belted mixed voice in the higher passages; slight grit
· Despite being richly descriptive, the caption completely misidentifies the primary voice gender and invents non-existent layered backing vocals.
🎧 Listen — interactive demo
Play each clip, then read what the base model, the full fine-tune and the frozen-encoder checkpoint
each say about it, with the Gemini reference caption underneath. The ten span genre families and the
solo / backing / instrument flags including cases the models get wrong — ten flattering clips
would make a prettier page and a useless one.
Measured on those ten:
- The base model describes, but it describes the wrong thing. The claim that once stood here — nine of ten base outputs are a lyric transcript — was measured against
openai/whisper-small, which is not the base of anything in this repo, and is withdrawn. Measured on all 158 held-out clips, 158 of 158 laion/music-whisper outputs (100 %) are music description; 0 are lyric transcripts and 0 are degenerate. The problem is that it describes an imagined band — tempo, drums, production — over clips that are one unaccompanied voice.
- Voice gender is right in 9 of 10. The miss is
opera_soprano_coloratura.
- One clip exposes a generation failure rather than a captioning one: on
metal_symphonic_soprano the prompt asked for a female voice, and the reference caption and
both fine-tunes independently hear a male tenor. LeVo 2 ignored the gender token; the captioners
are right.
The two checkpoints
Both live in this repo as subfolders. Load with subfolder=:
1from transformers import WhisperProcessor, WhisperForConditionalGeneration
2
3sub = "full" # or "frozen-encoder"
4proc = WhisperProcessor.from_pretrained("TTS-AGI/vocal-music-whisper", subfolder=sub)
5model = WhisperForConditionalGeneration.from_pretrained("TTS-AGI/vocal-music-whisper", subfolder=sub)
| full | frozen-encoder |
|---|
| What moves | encoder and decoder | decoder only, encoder frozen |
| Trainable params | 241M / 242M | 154M / 242M |
| Learning rate | 1e-05 | 5e-05 |
| Result | val loss 3.2411 → 1.1080 (65.8% lower), 0.13 GPU-h | val loss 3.2411 → 0.9461 (70.8% lower), 0.14 GPU-h |
Superseded — these are the v1 checkpoints. The default is now the repo root (v2/unbalanced-lr3e5), which beats both of these on caption quality, validation loss and voice gender. The comparison below is kept because it is why every later run freezes the encoder.
Of the two v1 checkpoints, frozen-encoder is the better one.
full adapts the acoustic front end as well as the caption language model, so it fits dry
unaccompanied voice best — but it is also the one that can drift away from the base model's
general music knowledge. frozen-encoder cannot forget anything the encoder knows, by
construction: the audio representation is bit-identical to the base model's and only the
decoder's captioning behaviour changes. If you intend to caption a mix of a cappella and
full-band music with a single model, prefer frozen-encoder. If you only ever see solo
voice, full is the better fit.
Why these learning rates
Whisper-small supervised fine-tuning conventionally sits at 1e-5 (HF's Whisper
fine-tuning recipes use 1e-5 down to 6.25e-6; Whisper's own paper pretrains small at
~5e-4, an order of magnitude above any fine-tune). On a corpus of this size, 5e-5+ applied
to the whole model is enough to overwrite the general music-captioning behaviour we
explicitly wanted to keep — so full uses 1e-5 with 10% warmup and cosine decay, which is
the standard "adapt without erasing" setting.
frozen-encoder uses 5e-5 precisely because far fewer parameters are in play: with
the encoder held fixed, the same nominal LR moves a much smaller, better-conditioned
subspace, and the risk it carries is decoder overfitting rather than representational
collapse. Same-LR comparisons between a full and a frozen fine-tune are not
apples-to-apples; matching each to its own parameter count is.
Training data
Two datasets, combined and then downsampled to balance:
All clips are LeVo 2 gen_type: vocal generations captioned by Gemini 3 Flash from the
audio, with every call asserted to carry a non-zero AUDIO prompt-token entry so no fabricated
caption could enter the set.
The fix that motivated the retrain
The male skew was not the captioner. A controlled probe (36 clips per arm, same genres and
seeds, only the position of the gender word changed) found it is prompt word order in the
generator: with the gender word first, 11.1 % of clips come out female; with the same word
last, 47.2 %. Confirmed acoustically at median F0 324 Hz against 276 Hz.
4,800 new clips were generated with the winning gender_last variant. The honest yield
was 36.3 %, not the 47.2 % the small probe suggested —
a second, larger probe of the same variant measured 36.1 %, which is what the full run
reproduced. The prompt fix roughly doubles female yield and does not solve the problem outright.
The balanced set was then drawn from the combined pool with no oversampling (every clip
appears at most once), by stratified downsampling of the male side genre by genre with a floor
of 12 per genre, so all 85 genres survive. Male clips were selected by
deterministic id hash, so the choice is reproducible and uncorrelated with caption length or
quality. Clips the judge could not gender were dropped rather than assigned a side.
Two instruments, and they disagree. The audio judge and a caption-derived gender label agree
per clip 84.8 % of the time yet differ ~3× on the corpus-wide female base rate. The
balance above is by the audio judge. Treat both as instruments, not ground truth.
The validation split
158 clips, held out from every genre bucket with a fixed seed before any training, and
unchanged across both rounds — so every val loss on this page is comparable to the
3.2411 base figure and to the previous release. No newly generated clip entered it: the val
split predates the rebalancing run and is 22 % female, which is why balanced
accuracy rather than raw accuracy is the headline.
Hyperparameters
| full | frozen-encoder |
|---|
| Base | laion/music-whisper (whisper-small topology, 12+12 layers, d=768) | same |
| Epochs | 3 | 3 |
| Batch (per GPU × GPUs) | 8 × 1 = 8 | 8 × 1 = 8 |
| Optimizer | AdamW β=(0.9, 0.98), eps 1e-6, wd 0.01 | same |
| Schedule | cosine, 10% linear warmup | same |
| Grad clip | 1.0 | same |
| Precision | bf16 autocast | same |
| Max target tokens | 448 | same |
| GPU-hours | 0.131 | 0.136 |
| Hardware | 1x GH200 (aarch64) per run; 4 runs concurrently on one node | same |
Loss curves
full — train: 25:3.255 · 300:1.686 · 600:1.320 · 875:1.242 · 1150:1.142 · 1450:1.126 · 1725:1.071 · 2025:1.097 · 2300:1.050 · 2575:1.024 · 2875:1.019 · 3150:1.034
full — val: 0:3.241 · 1053:1.217 · 2106:1.121 · 3159:1.108
frozen-encoder — train: 25:3.173 · 300:1.316 · 600:1.096 · 875:1.055 · 1150:0.930 · 1450:0.924 · 1725:0.867 · 2025:0.888 · 2300:0.792 · 2575:0.775 · 2875:0.765 · 3150:0.772
frozen-encoder — val: 0:3.241 · 1053:1.033 · 2106:0.950 · 3159:0.946
(step : loss)
Data-mix ablation
Gemini flagged a substantial minority of clips as carrying audible instruments despite
every prompt asking for none, so "what is actually in the training mix" is a real axis
rather than a cosmetic one. The same two variants were therefore also trained on the
subset with instruments_audible == false. The validation set is identical and
unfiltered in every cell, so all four val losses are directly comparable.
| Run | Variant | Mix | Train clips | Val before | Val (best, published) | Best epoch | GPU-h |
|---|
full_all | full | all | 8,424 | 3.2411 | 1.1080 | 3 | 0.131 |
frozen_all | frozen | all | 8,424 | 3.2411 | 0.9461 | 3 | 0.136 |
full_vocalsonly | full | vocalsonly | 5,431 | 3.2411 | 1.2605 | 3 | 0.092 |
frozen_vocalsonly | frozen | vocalsonly | 5,431 | 3.2411 | 1.0992 | 2 | 0.074 |
Two results fall out of this, and the second was not the expected one:
- Freezing the encoder wins.
frozen-encoder reaches a lower val loss than the full
fine-tune on both data mixes, while being structurally incapable of forgetting what the
encoder knows about general music.
- Filtering the training data to "vocals only" makes it worse, not better. Dropping the
2,993
clips Gemini flagged as carrying audible instruments costs roughly 0.15 val loss in both
variants. The val set is unfiltered and reflects the real distribution, so a model that
never saw a backing instrument is simply worse at describing the clips that have one.
The intuitive clean-up hurt.
Before / after on held-out val clips
tr_bulgarian_diaphonic_evo_s2 — genre bulgarian_diaphonic
Gemini reference: This recording features a solo male tenor voice performing a pop-rock arrangement, accompanied by layered vocal harmonies during the chorus sections. The singer begins the verse in a warm, resonant mid-register with a legato phrasing and a relatively straight tone, punctuated by subtle glottal attacks for emphasis. As the track progresses into the chorus, the vocal style shifts towards a powerful belt, utilizing a well-support…
Base laion/music-whisper: The listener hears a recording dominated by a male vocalist. The vocal performance is characterized by a clear and slightly nasal timbre, conveying a sense of sincerity. The vocal delivery is the primary focus, with no other non-lyrical vocal sounds present. The song's instrumentation is minimal, consisting solely of a simple acoustic guitar accompaniment. The tempo is slow, contributing to the overall mood. The genre is class…
full: This recording features a solo male tenor voice that begins with an intimate, breathy delivery before transitioning into a powerful, multi-layered vocal arrangement. The genre is firmly rooted in Indie Pop, characterized by its soulful and uplifting emotional arc. The singer employs a warm, resonant timbre with a slight grit that adds a sense of vulnerability to the performance. As the track progresses, the vocalist transition…
frozen-encoder: This recording features a solo male tenor voice that begins with an intimate, breathy delivery before expanding into a rich, multi-tracked arrangement with layered harmonies and doubled vocal lines. The genre is contemporary pop-rock, characterized by a soulful and emotive vocal style. The singer employs a mix of chest and head voice, transitioning into a powerful belt during the more intense passages. His technique includes s…
levo2__metal_s0 — genre metal
Gemini reference: This recording features a powerful male tenor performing an anthemic arena rock passage, beginning with a gritty solo vocal that eventually expands into a layered, multi-part harmony. The singer employs a robust chest-dominant mix and high-energy belting, characterized by a warm yet textured timbre with a natural rasp that adds emotional weight. His delivery is marked by sustained legato lines and deliberate glottal attacks, n…
Base laion/music-whisper: The listener hears a track that immediately establishes a strong rhythmic foundation with a driving drum beat and a prominent bassline. The tempo is approximately 120 beats per minute, contributing to the song's energetic feel. The instrumentation is primarily electronic, featuring a distorted synth bass providing a heavy low-end presence, alongside a driving drum beat that propels the music forward. A male vocalist takes the …
full: This recording features a powerful male tenor performing in a dramatic musical theatre style, beginning with a solo lead that is later joined by lush, multi-tracked backing harmonies. The singer employs a robust mixed-voice technique, transitioning into a resonant belt with a controlled, medium-width vibrato and occasional glottal attacks for emotional emphasis. His timbre is warm and rich, characterized by a slight grit and a…
frozen-encoder: This recording features a powerful male tenor performing in a contemporary alternative rock style, beginning with a solo vocal that soon expands into a rich, multi-layered arrangement with doubled voices and soaring harmony parts. The singer employs a robust mixed-voice technique, transitioning into intense belting with a controlled, medium-width vibrato and occasional glottal attacks that add emotional weight. His timbre is w…
g60_metal_black_018 — genre metal_black
Gemini reference: This high-energy alternative rock performance features a powerful solo male lead complemented by rich, multi-tracked backing harmonies that provide a dense, anthemic texture. The vocalist utilizes a robust belting technique with a gritty, distorted edge, characteristic of post-grunge aesthetics. His delivery is marked by a fast, wide vibrato and aggressive glottal attacks, transitioning seamlessly between a resonant chest-domi…
Base laion/music-whisper: The listener hears a track that immediately establishes a strong rhythmic foundation with a driving drum beat and a prominent bassline. The tempo is approximately 140 beats per minute, contributing to the song's energetic feel. The instrumentation is characterized by a prominent drum kit, featuring a punchy kick drum, a snare drum, and a hi-hat, providing a solid rhythmic backbone. A male vocalist takes the lead, delivering th…
full: This recording features a powerful male solo voice that transitions into a rich, multi-layered arrangement with layered backing harmonies and unison doubling. The genre is a dramatic musical theater-rock, characterized by a defiant and intense delivery. The singer employs a robust mixed voice with a resonant, warm timbre, utilizing a strong chest-dominant mix and occasional glottal attacks for emotional emphasis. The vocal ran…
frozen-encoder: This recording features a powerful male solo voice that transitions into a layered, anthemic arrangement with doubled vocals and harmonies during the climactic sections. The performance sits firmly within the alternative rock genre, characterized by its defiant and anthemic energy. The singer employs a robust belting technique, utilizing a strong mixed voice to reach the higher register with a slight, intentional grit. His tim…
g60_opera_tenor_dramatic_056 — genre opera_tenor_dramatic
Gemini reference: This recording features a solo male lead, likely a high baritone or tenor, who begins with a soulful, contemplative delivery before being joined by a powerful, multi-layered gospel-style choir. The genre leans heavily into contemporary gospel with folk-rock sensibilities, characterized by a transition from intimate storytelling to a triumphant choral anthem. The lead vocalist employs a resonant chest-mix with a moderate vibrat…
Base laion/music-whisper: The listener hears a piece of music primarily characterized by a female vocalist. The vocal performance is delivered by an adult female, exhibiting a clear and slightly operatic timbre. The vocal quality is described as clear and resonant. The instrumentation is composed of a piano, strings, and a choir. The tempo of the music is slow, approximately 70 beats per minute. The genre of the music is classified as either Classical …
full: This recording features a powerful male tenor performing in a dramatic musical theater style, beginning with a solo voice that is later joined by lush, multi-tracked backing harmonies. The singer employs a resonant mixed voice with a warm, slightly gritty timbre, transitioning into a powerful belt as the melody ascends. His technique is characterized by a wide, controlled vibrato and precise legato phrasing, punctuated by occa…
frozen-encoder: This recording features a powerful male tenor performing in a contemporary musical theater style, beginning with a solo vocal that soon expands into a rich, multi-layered arrangement with lush harmonies and doubled parts. The singer employs a resonant mixed voice with a bright, forward placement, utilizing a consistent vibrato and smooth legato phrasing that transitions into a powerful belt in the upper register. The timbre is…
What improved
- No more phantom instruments. The base model reliably invents a band — "gentle acoustic
guitar", "soft piano accompaniment", "steady drum groove" — under clips that are literally
one unaccompanied voice. Both fine-tunes stop doing that on this domain.
- Voice-count awareness. The captions now distinguish a genuinely solo take from one with
doubled/harmonised/choir backing, which is the single most useful label in a cappella data
and the axis the dataset was built around.
- Vocal vocabulary. Register, tessitura, vibrato width, belt vs mixed vs head voice,
glottal onsets, melisma — the base model rarely reaches for these; the fine-tunes do.
- Format consistency. Output is flowing prose of predictable length rather than the base
model's mix of prose and "well-suited for…" ad copy.
What regressed
Reported honestly, because both of these are real:
- Instrumentation description is degraded outside this domain. The training targets almost
never contain instruments, so
full in particular has learned that instruments are usually
absent. On full-band music it under-describes or omits the backing. frozen-encoder is
markedly more robust here — the encoder is untouched — which is why it is the safer general
pick.
- Genre calls inherit LeVo 2's failure modes. The training audio is synthetic, and for nine
genres LeVo 2 provably cannot produce the requested style (see the dataset card). Captions
for those clips describe what was generated, not what was asked for. The models will
reproduce that mapping.
- Synthetic-audio bias. Every training clip is a neural generation. Expect some
transfer gap to real human a cappella recordings; the recording-character sentences in
particular describe LeVo 2's rendering characteristics.
- Tempo/BPM and key statements are less reliable. They were rarely present in the
targets, so they got no reinforcement.
Intended use
Captioning and tagging of solo-voice / a cappella audio: dataset curation, retrieval,
filtering for singing-voice corpora. Use the
repo root unless you are deliberately
comparing checkpoints.
Not a general music captioner — use the base
laion/music-whisper for that, and use it
too if voice gender is what you need.
Reproducing
Everything below is in
code/ in this repo and runs as written.
Caption your own audio — files, globs or a directory; the default checkpoint is the repo
root, so no --subfolder is needed:
1pip install "transformers>=4.40" torch soundfile scipy
2python code/caption.py song.wav
3python code/caption.py clips/ --batch 8 --jsonl captions.jsonl
4python code/caption.py song.wav --model TTS-AGI/vocal-music-whisper --subfolder v2/balanced-lr1e4 # an ablation
or in four lines of Python:
1import soundfile as sf
2from transformers import WhisperForConditionalGeneration, WhisperProcessor
3
4proc = WhisperProcessor.from_pretrained("TTS-AGI/vocal-music-whisper")
5model = WhisperForConditionalGeneration.from_pretrained("TTS-AGI/vocal-music-whisper").eval()
6model.config.forced_decoder_ids = None # or it transcribes lyrics instead
7model.generation_config.forced_decoder_ids = None
8
9x, sr = sf.read("song.wav") # must be mono 16 kHz
10feats = proc(x, sampling_rate=16000, return_tensors="pt").input_features
11print(proc.batch_decode(model.generate(feats, max_new_tokens=440, num_beams=1),
12 skip_special_tokens=True)[0])
max_new_tokens=440 matters: these captions run 130–190 words, and the 200-token cap used
in an earlier revision cut a third of them off mid-sentence. 440 is the ceiling — Whisper's
max_target_positions is 448 and the three forced start tokens count against it.
Train it. One GPU per run; the four v2 runs fit on one 4-GPU node
(code/retrain.sbatch is the exact Slurm script, including the 150-second stagger between
launches that stops concurrent import torch off a shared filesystem from wedging every
worker):
1python code/finetune.py --variant frozen --mix all --lr 3e-5 \
2 --data train.jsonl --val val.jsonl --epochs 10 --bs 8 \
3 --tag unbalanced-lr3e5 --out ckpt/unbalanced-lr3e5 # <- the default checkpoint
Regenerate the held-out captions (code/capgen.sbatch runs all four checkpoints across
four GPUs):
1python code/eval_gender_val.py --name orig_lr3e5 --path ckpt/unbalanced-lr3e5 \
2 --max-new-tokens 440 --out capgen_orig_lr3e5.json
Re-judge caption quality (needs a Gemini key in HYPRLAB_API_KEY; ~$5 and ~10 minutes
for all 1,264 judgements, and it resumes from the JSONL if interrupted):
1python code/judge_caption.py \
2 --preds base=capgen_base.json frozen_all_3ep=capgen_frozen_all_3ep.json \
3 orig_lr3e5=capgen_orig_lr3e5.json bal_lr1e4=capgen_bal_lr1e4.json \
4 --out capjudge.jsonl --workers 16 --rep 0
5python code/judge_caption.py ... --rep 1 # second independent pass
6python code/capjudge_summary.py --jsonl capjudge.jsonl --out capjudge_summary.json
Re-measure voice gender on the same split:
1python code/eval_gender_val.py --name orig_lr3e5 --path ckpt/unbalanced-lr3e5
2python code/evalg_summary.py
Related
The generator. The audio is LeVo 2 / SongGeneration v2. laion/moss-tts-local-transformer-4.55b-voice-acting-v2 is not the base model for anything here — that is a
speech/voice-acting model and is unrelated to this music corpus.
This project.
Scoring and captioning models.
Citation
1@misc{vocalmusicwhisper,
2 title = {Vocal Music Whisper: a cappella-specialised music captioning},
3 author = {LAION},
4 year = {2026},
5 url = {https://huggingface.co/TTS-AGI/vocal-music-whisper}
6}
Base model laion/music-whisper (CC-BY-4.0). Training data TTS-AGI/levo2-vocals-captioned. Released under CC-BY-4.0.
