Whisper-Small Dutch - High-Quality Filtered Synthetic Data
This model is a fine-tuned version of openai/whisper-small for Dutch automatic speech recognition (ASR). It was trained on Common Voice 17.0 Dutch combined with WAVe-filtered synthetic speech data using a strict high-quality threshold (q ≥ 0.8).
Introduction
How the Data Was Created
The training data combines real speech from Common Voice 17.0 with synthetic speech generated through a two-stage pipeline:
Transcript Generation: We used GPT-4o-mini to generate Dutch transcripts that match the word count distribution observed in Common Voice, ensuring realistic utterance lengths and diverse linguistic content.
Speech Synthesis: Each transcript was converted to audio using OpenAI's TTS-1 model with 9 different voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer), producing 34,898 synthetic samples.
Quality Filtering with WAVe: Raw synthetic speech often contains defects such as mispronunciations, omitted words, or prosodic anomalies. To address this, we applied WAVe (Word-Aligned Verification), a model that assesses audio-text alignment at the word level rather than the sentence level. WAVe uses multi-head attention to align each word to its corresponding audio frames and assigns per-word confidence scores via a GLU-based scorer. For this model, only samples scoring above the strict threshold (q ≥ 0.8) were retained, resulting in 10,555 high-quality synthetic samples.
How the Model Was Created
The model was fine-tuned from openai/whisper-small using the Hugging Face Transformers library with the following approach:
Mixed Training: Combined 34,952 real speech samples from Common Voice 17.0 Dutch with 10,555 strictly WAVe-filtered synthetic samples (45,507 total).
Optimization: Trained for 5 epochs with a learning rate of 1e-5, global batch size of 256, and BF16 precision on an NVIDIA H200 GPU.
Checkpoint Selection: The best checkpoint was selected based on validation loss, occurring at step 350 with a validation loss of 0.1493.
This high-quality filtering approach achieves 35% reduction in training steps compared to using all synthetic data, while maintaining competitive ASR performance.
Model Details
Property
Value
Base Model
openai/whisper-small
Language
Dutch (nl)
Task
Automatic Speech Recognition (transcribe)
Parameters
244M
Training Data
Common Voice 17.0 + High-Quality Synthetic (q ≥ 0.8)
Total Training Samples
45,507
Sampling Rate
16kHz
Evaluation Results
This Model (whisper-small-high-mixed-nl)
Metric
Value
Validation Loss
0.1493
Validation WER
8.76%
Test WER (Common Voice)
11.00%
Test WER (MLS)
29.91%
Best Checkpoint
Step 350
Max Training Steps
890
Comparison with Other Training Configurations (Whisper-Small Dutch)
Training Data
Max Steps
Val Loss
Val WER
Test WER (CV)
Test WER (MLS)
Common Voice Only
680
0.1491
8.73%
11.13%
30.71%
High-Quality Filtered + CV
890
0.1493
8.76%
11.00%
29.91%
Mid-High Quality Filtered + CV
1,270
0.1484
8.73%
10.86%
30.04%
All Synthetic + CV (Unfiltered)
1,365
0.1484
8.64%
10.91%
30.06%
Key Performance Highlights
Most efficient training: Only 890 max steps (35% fewer than unfiltered)
1.2% relative improvement on Common Voice test set vs baseline (11.00% vs 11.13%)
2.6% relative improvement on MLS benchmark vs baseline (29.91% vs 30.71%)
Best quality-to-compute ratio: Achieves strong results with minimal synthetic data
