You can deploy this model and start earning money today!

laion/
voicenet-dimension-predictors-commercial

Name: voicenet-dimension-predictors-commercial
Availability: InStock

Views

No views yet

VoiceNet Dimension Predictors (VoiceCLAP-commercial)

57 per-dimension VoiceNet predictors on top of frozen VoiceCLAP-commercial (768-d, L2-normalized) audio embeddings. For every one of the 57 VoiceNet perceptual voice dimensions this repo ships two lightweight MLP heads:

a regression head (continuous score on the dimension's 0-k scale), and
a classification head (discrete level/bucket + confidence).

One commercial encode -> all 57 dimensions. Embed a clip once with the bundled VoiceCLAP-commercial encoder, then run all 114 heads (57x2) on that single 768-d vector.

This repo is self-contained: it bundles the full VoiceCLAP-commercial embedder under voiceclap_commercial/, so no separate model download is needed.

How to read these numbers (plain language)

Each dimension is scored on an ordinal 0-k scale (k=6 for most, so 0..6; a few have fewer levels). We report, on a held-out validation set the model never trained on:

MAE (regression) - Mean Absolute Error: on average, how many scale-points the predicted score is away from Gemini's score. Lower is better. MAE 0.5 means predictions land about half a level off on a 0-6 scale; MAE 1.3 means more than a full level off.
Pearson r (regression) - how well predictions track the true ranking (correlation), from -1 to 1. Higher is better. ~0.9 = predictions move almost in lockstep with the labels; ~0.5 = only a weak trend.
within-1 (regression) - the fraction of clips whose predicted score is within +-1 level of the true score. Higher is better (0.90 = 90% of predictions are at most one level off).
accuracy (classification) - fraction of clips whose predicted discrete bucket exactly matches Gemini's. Higher is better. Chance is ~1/k, so ~0.14 for a 7-level axis; 0.60 is far above chance.

So a strong dimension looks like low MAE + high Pearson + high within-1 + high accuracy.

What are the dimensions?

The 57 VoiceNet dimensions cover arousal, valence, gender, age, tempo, brightness, resonance (chest/head/nasal/...), speaking styles (news, ASMR, dramatic, ...), recording quality, and more. See dimensions.py for every code, human name, level count, and per-level definition.

How it was trained (round-1+2+3, best-variant-per-dimension)

Labels: laion/emolia-voicenet-gemini-annotations - Emolia clips each re-scored on their single VoiceNet dimension by Gemini-3.5-flash. This retrain uses the combined round-1 + round-2 + round-3 labels (round 3 topped up the most data-starved score levels of the hardest dimensions).
Frozen validation split: the same held-out validation ids as the previously published run (val_ids.json), so every number below is directly comparable to the round-1+2 release.
Embeddings: each unique clip encoded once with VoiceCLAP-commercial (768-d, L2-normalized).
Three balancing variants per dimension - cap3x / cap4x / cap5x. Samples are bucketed by Gemini score; each bucket is capped at 3x / 4x / 5x of S (S = the count of the second-smallest usable bucket), larger buckets randomly subsampled to that cap. Near-empty buckets (<4) are skipped. The validation split is frozen and identical across variants.
Heads / losses: regression = HuberLoss (delta=1.5), H=64, dropout=0.33; classification = CrossEntropy, H=48, dropout=0.33. Adam, 50 epochs, input standardized by train mean/std, best-val checkpoint kept (val MAE for reg, val accuracy for cls).
Best-variant-per-dimension selection: for each dimension we keep the variant that did best on the frozen val set - the regression head with the lowest val MAE, and the classification head with the highest val accuracy (chosen independently). Those are the top-level regression/ + classification/ defaults that predict.py loads. All 6x57 checkpoints are also shipped under variants/{cap3x,cap4x,cap5x}/ for reproducibility.

Variant comparison (mean over all 57 dimensions)

variant	mean reg MAE	mean reg Pearson	mean cls acc	mean cls within-1	mean n_train
cap3x	0.767	0.78	0.603	0.859	1562
cap4x	0.762	0.79	0.595	0.857	1906
cap5x	0.756	0.79	0.592	0.853	2188

Best-variant picks (how many dimensions selected each variant):

Regression (min val MAE): cap3x: 18, cap4x: 15, cap5x: 24
Classification (max val acc): cap3x: 28, cap4x: 15, cap5x: 14

The best-per-dimension defaults (mixing variants) reach mean reg MAE 0.744, mean Pearson 0.79, mean within-1 0.870, mean cls accuracy 0.617, mean cls within-1 0.863.

Did round-3 data help? (before / after)

Against the previous round-1+2 published defaults (regression = cap4x, classification = cap2x), on the identical frozen validation split:

metric	round-1+2 default	round-1+2+3 best-per-dim	change
mean reg MAE	0.768	0.744	+0.024 (better)
mean cls accuracy	0.593	0.617	+0.025 (better)

Top-5 dimensions most improved on MAE (per-dim delta vs the round-1+2 cap4x regression MAE): R_MIXD (Mixed Resonance, 0.920→0.787, Δ0.133), R_NASL (Nasal Resonance, 0.780→0.683, Δ0.096), COGL (Cognitive Load, 1.131→1.052, Δ0.079), R_THRT (Throat Resonance, 0.797→0.718, Δ0.078), S_CART (Cartoonish Style, 1.191→1.115, Δ0.076).

Per-dimension metrics (default = best variant, sorted by MAE)

dim	name	k	reg variant	val MAE	Pearson	within-1	cls variant	cls acc	n_train
`EXPL`	Content Appropriateness (3-point Scale)	3	cap3x	0.444	0.42	0.939	cap3x	0.697	724
`R_HEAD`	Head Resonance	7	cap5x	0.478	0.88	0.957	cap3x	0.757	1411
`BKGN`	Background Noise	5	cap3x	0.493	0.83	0.967	cap3x	0.650	3595
`SMTH`	Smoothness	7	cap3x	0.499	0.89	0.986	cap5x	0.775	1426
`VOLT`	Volatility	7	cap3x	0.516	0.95	0.981	cap4x	0.571	3559
`R_ORAL`	Oral Resonance	7	cap5x	0.532	0.85	0.937	cap3x	0.698	1086
`RANG`	Pitch Range	7	cap4x	0.533	0.87	0.938	cap3x	0.675	2015
`ATCK`	Attack	7	cap3x	0.537	0.93	0.935	cap4x	0.645	1205
`S_FORM`	Formal Style	7	cap4x	0.557	0.89	0.922	cap3x	0.689	3615
`R_MASK`	Mask Resonance	7	cap4x	0.562	0.83	0.961	cap3x	0.745	656
`R_CHST`	Chest Resonance	7	cap4x	0.562	0.87	0.960	cap3x	0.627	2280
`STRU`	Structure	7	cap4x	0.573	0.90	0.952	cap5x	0.590	2146
`CLRT`	Articulation Clarity	7	cap5x	0.575	0.88	0.902	cap5x	0.720	802
`HARM`	Harmonicity	7	cap5x	0.581	0.79	0.917	cap5x	0.683	1090
`TEMP`	Tempo	7	cap4x	0.582	0.91	0.929	cap3x	0.600	1861
`CHNK`	Chunking	7	cap3x	0.596	0.90	0.904	cap4x	0.614	902
`BRGT`	Brightness	7	cap5x	0.601	0.77	0.937	cap3x	0.651	476
`AROU`	Arousal	7	cap5x	0.601	0.93	0.943	cap5x	0.629	4514
`GEND`	Perceived Gender	7	cap5x	0.612	0.89	0.949	cap3x	0.705	1711
`AGEV`	Voice Age	7	cap4x	0.622	0.89	0.881	cap4x	0.702	1329
`REGS`	Register	7	cap3x	0.629	0.88	0.917	cap4x	0.679	998
`METL`	Metallic Character	7	cap5x	0.643	0.66	0.896	cap4x	0.729	615
`ROUG`	Roughness	7	cap5x	0.646	0.76	0.933	cap5x	0.600	239
`ESTH`	Esthetics	7	cap5x	0.647	0.80	0.893	cap3x	0.587	2497
`STNC`	Stance	7	cap3x	0.660	0.88	0.931	cap3x	0.657	3140
`S_CASU`	Casual Style	7	cap3x	0.667	0.92	0.915	cap3x	0.691	2275
`RCQL`	Recording Quality	7	cap3x	0.667	0.82	0.933	cap3x	0.560	2073
`S_AUTH`	Authoritative Style	7	cap5x	0.677	0.83	0.844	cap3x	0.589	4395
`R_NASL`	Nasal Resonance	7	cap5x	0.683	0.83	0.918	cap5x	0.694	1083
`RESP`	Respiration	7	cap3x	0.699	0.88	0.942	cap3x	0.581	878
`R_THRT`	Throat Resonance	7	cap5x	0.718	0.72	0.883	cap5x	0.633	539
`WARM`	Warmth	7	cap3x	0.739	0.72	0.897	cap4x	0.603	1742
`VULN`	Vulnerability	7	cap3x	0.742	0.81	0.895	cap3x	0.611	1822
`EMPH`	Emphasis	7	cap4x	0.762	0.87	0.873	cap3x	0.569	2019
`S_DRAM`	Dramatic Style	7	cap5x	0.765	0.87	0.905	cap3x	0.590	2288
`FULL`	Fullness	7	cap4x	0.766	0.73	0.910	cap5x	0.582	172
`VALN`	Valence	7	cap3x	0.774	0.79	0.894	cap4x	0.529	1454
`S_NARR`	Narrator Style	7	cap5x	0.776	0.86	0.864	cap4x	0.523	4242
`R_MIXD`	Mixed Resonance	7	cap5x	0.787	0.39	0.864	cap5x	0.864	106
`VALS`	Valence Shift	7	cap4x	0.791	0.80	0.812	cap4x	0.565	1831
`DARC`	Dynamic Arc	7	cap5x	0.807	0.83	0.839	cap4x	0.634	1599
`TENS`	Tension	7	cap4x	0.814	0.77	0.852	cap3x	0.568	925
`S_NEWS`	Newsreader Style	7	cap4x	0.823	0.71	0.815	cap3x	0.554	2429
`S_CONV`	Conversational Style	7	cap3x	0.824	0.79	0.892	cap4x	0.516	2449
`DFLU`	Disfluency	7	cap4x	0.844	0.81	0.839	cap5x	0.419	2079
`S_PLAY`	Playful Style	7	cap5x	0.847	0.79	0.827	cap3x	0.587	4607
`S_MONO`	Monologue Style	7	cap5x	0.893	0.73	0.774	cap5x	0.505	6405
`S_STRY`	Storytelling Style	7	cap5x	0.905	0.84	0.778	cap3x	0.656	4361
`S_WHIS`	Whisper-Talk Style	7	cap5x	0.947	0.81	0.817	cap3x	0.463	1460
`COGL`	Cognitive Load	7	cap3x	1.052	0.61	0.734	cap3x	0.543	1354
`S_TECH`	Teacher/Didactic Style	7	cap4x	1.104	0.69	0.691	cap4x	0.568	446
`S_CART`	Cartoonish Style	7	cap4x	1.115	0.68	0.683	cap4x	0.524	1399
`FOCS`	Focus	7	cap3x	1.115	0.65	0.724	cap3x	0.663	2216
`VFLX`	Velocity Flux	7	cap3x	1.231	0.56	0.656	cap3x	0.441	1393
`S_RANT`	Ranting/Angry Style	7	cap5x	1.238	0.72	0.654	cap5x	0.568	241
`ARSH`	Arousal Shift	7	cap5x	1.257	0.53	0.633	cap4x	0.561	3575
`S_ASMR`	ASMR Style	7	cap5x	1.316	0.68	0.667	cap5x	0.565	418

Strongest (lowest MAE): EXPL, R_HEAD, BKGN, SMTH, VOLT.
Weakest (highest MAE): FOCS, VFLX, S_RANT, ARSH, S_ASMR.

The weakest axes are the fine resonance/timbre and shift/flux dimensions (e.g. R_MIXD, R_NASL, arousal/valence-shift, velocity-flux): the Gemini labels for these are heavily collapsed into one or two buckets (VoiceCLAP zero-shot bucketing cannot separate them), so there is little balanced signal to learn and their errors are inherently larger. Coarse perceptual axes (arousal, valence, gender, formality, tempo, brightness, background noise) predict well.

Inference

python

from predict import VoiceNetPredictor
p = VoiceNetPredictor()             # bundled embedder + best-per-dim heads (114 total)
out = p.predict("audio.wav")        # one encode -> 57 dims
print(out["AROU"])   # {'name': 'Arousal', 'reg_score': .., 'cls_bucket': .., 'confidence': .., ...}

# force one balancing variant instead of the best-per-dim defaults:
p4 = VoiceNetPredictor(variant="cap4x")

or from the shell: python predict.py audio.wav (add --variant cap4x to force a variant).

Each dimension returns the regression score, the classification bucket, and the softmax confidence.

Files

regression/<DIM>.pt, classification/<DIM>.pt - 57 + 57 best-per-dimension default heads. Each: {state_dict, mu, sd, arch, dim, name, levels, val_metrics}.
variants/{cap3x,cap4x,cap5x}/{regression,classification}/<DIM>.pt - all 6x57 variant heads.
voiceclap_commercial/ - the full bundled VoiceCLAP-commercial embedder (self-contained).
dimensions.py, predict.py, requirements.txt.
metrics.parquet - per dimension x variant validation metrics.
metrics_best_per_dim.parquet - the chosen best variant per dimension per head + its metrics.

Notes & limitations

Labels are Gemini-3.5-flash perceptual estimates from a single flash-LLM annotator, not ground truth.
Fine resonance/timbre axes are collapsed in the labels and predict near-constant - expected.
The bundled embedder is included for self-containment; heads run on its 768-d L2-normalized output.

Project & Taxonomy

These predictors are the models of the VoiceNet project — predicting how speech is delivered across 57 perceptual voice/prosody dimensions (0–6).

GitHub project & docs: https://github.com/LAION-AI/voicenet — self‑contained documentation, including a full inference guide (throughput optimization, WebDataset/mp3 loading, multi‑GPU scaling) and training guide.
Live demo: https://projects.laion.ai/voicenet/ — these predictors on 100 diverse multilingual Emolia clips.
Taxonomy: every dimension's code, name, level count, and per‑level 0–6 definition ships in this repo as dimensions.py; the full narrative rubric is on GitHub as taxonomy/voicenet_taxonomy.md.

The VoiceNet taxonomy and project come from the forthcoming VoiceNet paper (not yet published).

License

CC-BY-4.0

laion/voicenet-dimension-predictors-commercial