[!WARNING]
Superseded by vibrato-v2.
A post-release audit found that v1's evaluation was contaminated: the model
trained on 18 of VocalSet's 20 singers with no singer-disjoint split, so the
accuracy figures below are effectively training-set numbers. Against measured
acoustic labels (Praat CPPS, jitter/shimmer, f0 deviation), v1's five quality
outputs all have negative R² — worse than predicting a constant. v2 fixes
the corpus coverage, roster, labels, and splits, and reports genuinely
held-out numbers. Use v2; v1 remains available for reproducibility.
Vibrato is an open vocal-AI model for the singing community, maintained by
Anycompany LLC. It powers Toney, a free iOS vocal-training app, and is
published for anyone to use, study, and improve.
The training pipeline (model.py, CoreML export) is included in this repository.
Roadmap: v1 (this release) — VocalSet-trained baseline. v2 — expanded
training data (augmentation + additional public datasets). v3 — federated
learning: on-device fine-tuning with privacy-preserving aggregated updates
(voices never leave the phone).
Intended Use
Real-time vocal analysis for singing practice and coaching. Designed to run on-device on iPhones via CoreML Neural Engine.
Primary use: Analyze vocal recordings to classify voice type, singing technique, vowel production, and vocal quality metrics.
Out of scope: Speech recognition, language identification, speaker identification, medical diagnosis of vocal disorders.
Model Architecture
Input: [1, 4000] raw audio (0.25s @ 16kHz)
-> SharedEncoder (4x ConvBlock with BatchNorm, stride-2 downsampling, AdaptiveAvgPool)
-> 256-dim feature vector
-> Voice Head (FC 256->128->6, logits — apply softmax at inference)
-> Technique Head (FC 256->128->7, logits — apply softmax at inference)
-> Vowel Head (FC 256->128->6, logits — apply softmax at inference)
-> Quality Head (FC 256->128->5, sigmoid applied in-model)
Training Data
Dataset
License
Size
Use
VocalSet
CC BY 4.0
10.1 hrs, 20 singers (18 used in v1 — see below)
All four heads (technique, vowel, voice type; quality labels heuristically derived from the audio)
Coverage in v1: Training used 1,562 of VocalSet's 3,613 files (43%). The
remainder were dropped because their technique folders were not in the
technique map (slow_piano, slow_forte, fast_forte, fast_piano, pp, forte,
messa, and the misspelled "vibrado" folder). Additionally, a gender-table
erratum in the singer roster (the code listed 11 female / 9 male singers,
while VocalSet actually contains 9 female / 11 male) caused singers male10
and male11 (364 files) to be silently skipped — so 18 of the 20 singers
contributed training data. Both issues are fixed in the v2 pipeline plan;
this card describes the shipped v1 model as trained.
Planned for v2 (not used in v1): Annotated-VocalSet (F0 annotations for pitch-stability labels), SVQTD (human vocal-quality labels).
Attribution: This model was trained on VocalSet, used under CC BY 4.0:
Julia Wilkins, Prem Seetharaman, Alison Wiss, Bryan Pardo, "VocalSet: A
Singing Voice Dataset", Proceedings of the 19th International Society for
Music Information Retrieval Conference (ISMIR), 2018. Zenodo DOI:
10.5281/zenodo.1193957.
Changes made to the original data for training: audio resampled to 16 kHz,
segmented into 0.25 s windows, technique folder labels remapped to the model's
7-class scheme, and heuristic quality labels added.
falsetto: 0 training samples reached this class in v1 (the folder-mapping
bug above dropped them), so its logit is effectively untrained — do not rely
on falsetto predictions.
nasal: trained on VocalSet's vocal_fry recordings (a labeling shortcut),
so it actually detects vocal fry, not nasality.
mixed: trained on lip_trill recordings, so it actually detects lip
trills, not mixed voice.
Voice types: VocalSet publishes no fach/voice-type metadata; the
soprano-through-bass labels are heuristic assignments made in this pipeline,
not ground truth.
strain: the heuristic pseudo-label saturates at 1.0 on ~83% of training
clips (the zero-crossing "jitter" estimate is not true pitch-period jitter),
so the strain output has limited dynamic range.
Evaluation
All numbers are on the held-out validation split (random 20% of windows) at
epoch 49. "Production condition" applies the peak normalization the iOS app
performs before inference — a preprocessing mismatch with training (which used
raw amplitudes), which measurably degrades accuracy; v2 will train and evaluate
under matched normalization.
Metric
Val (reported)
Val (production condition)
Majority-class baseline
Voice type accuracy
47.9%
—
~22.6%
Technique accuracy
48.2%
30.4%
25.6%
Vowel accuracy
57.8%
48.6%
20.8%
Quality MSE
0.0292
—
—
On the two singers never seen in training (male10/male11, excluded by the
roster bug): technique 48.9% / vowel 64.4% raw, dropping to 31.0% / 53.4%
under production normalization. The quality head reaches R-squared of
approximately 0.32 against its own deterministic heuristic targets.
Note that the validation split is random over 0.25 s windows, so windows from
the same recording (and singer) appear in both train and val — these numbers
are optimistic relative to true singer-held-out generalization.
Limitations
Trained on ~10 hrs from professional singers (VocalSet; 18 of its 20
singers due to the roster bug above) — limited demographic and stylistic
coverage; expect degraded accuracy on voice types, languages, and styles
outside that distribution.
0.25 s context window — no long-range phrase analysis.
Quality scores are heuristic-derived labels, not clinician annotations.
Not for speaker ID, speech recognition, or medical use.
Ethical Considerations
The model is not designed or evaluated for speaker identification
No personally identifiable information is stored or inferred
All inference runs on-device; no audio data is transmitted
The model is NOT a medical device and should not be used for vocal health diagnosis
