Whisper-Small Dutch - Mixed Synthetic Data (Mid-High Quality Filtered)
This model is a fine-tuned version of openai/whisper-small for Dutch automatic speech recognition (ASR). It was trained on Common Voice 17.0 Dutch combined with WAVe-filtered synthetic speech data (quality threshold q ≥ 0.5).
Introduction
How the Data Was Created
The training data combines real speech from Common Voice 17.0 with synthetic speech generated through a two-stage pipeline:
Transcript Generation: We used GPT-4o-mini to generate Dutch transcripts that match the word count distribution observed in Common Voice, ensuring realistic utterance lengths and diverse linguistic content.
Speech Synthesis: Each transcript was converted to audio using OpenAI's TTS-1 model with 9 different voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer), producing 34,898 synthetic samples.
Quality Filtering with WAVe: Raw synthetic speech often contains defects such as mispronunciations, omitted words, or prosodic anomalies. To address this, we applied WAVe (Word-Aligned Verification), a model that assesses audio-text alignment at the word level rather than the sentence level. WAVe uses multi-head attention to align each word to its corresponding audio frames and assigns per-word confidence scores via a GLU-based scorer. Samples scoring below the threshold (q < 0.5) were removed, retaining 30,182 high-quality synthetic samples.
How the Model Was Created
The model was fine-tuned from openai/whisper-small using the Hugging Face Transformers library with the following approach:
Mixed Training: Combined 34,952 real speech samples from Common Voice 17.0 Dutch with 30,182 WAVe-filtered synthetic samples (65,134 total).
Optimization: Trained for 5 epochs with a learning rate of 1e-5, global batch size of 256, and BF16 precision on an NVIDIA H200 GPU.
Checkpoint Selection: The best checkpoint was selected based on validation loss, occurring at step 500 with a validation loss of 0.1484.
This approach achieves the best Test WER (10.86%) among all Whisper-Small Dutch configurations while maintaining strong cross-domain generalization.
Model Details
Property
Value
Base Model
openai/whisper-small
Language
Dutch (nl)
Task
Automatic Speech Recognition (transcribe)
Parameters
244M
Training Data
Common Voice 17.0 + Mid-High Quality Synthetic (q ≥ 0.5)
Total Training Samples
65,134
Sampling Rate
16kHz
Evaluation Results
This Model (whisper-small-mixed-cv-nl)
Metric
Value
Validation Loss
0.1484
Validation WER
8.73%
Test WER (Common Voice)
10.86%
Test WER (MLS)
30.04%
Best Checkpoint
Step 500
Max Training Steps
1,270
Comparison with Other Training Configurations (Whisper-Small Dutch)
Training Data
Max Steps
Val Loss
Val WER
Test WER (CV)
Test WER (MLS)
Common Voice Only
680
0.1491
8.73%
11.13%
30.71%
High-Quality Filtered + CV
890
0.1493
8.76%
11.00%
29.91%
Mid-High Quality Filtered + CV
1,270
0.1484
8.73%
10.86%
30.04%
All Synthetic + CV (Unfiltered)
1,365
0.1484
8.64%
10.91%
30.06%
Key Performance Highlights
Best Test WER (10.86%) on Common Voice among all Whisper-Small Dutch configurations
2.4% relative improvement on Common Voice test set vs baseline (10.86% vs 11.13%)
2.2% relative improvement on MLS benchmark vs baseline (30.04% vs 30.71%)
7% fewer training steps than unfiltered synthetic data while achieving better in-domain performance
