Whisper-Large-v3 Dutch - High-Quality Filtered Synthetic Data
This model is a fine-tuned version of openai/whisper-large-v3 for Dutch automatic speech recognition (ASR). It was trained on Common Voice 17.0 Dutch combined with WAVe-filtered high-quality synthetic speech data only using a strict threshold (q ≥ 0.8).
Introduction
How the Data Was Created
The training data combines real speech from Common Voice 17.0 with synthetic speech generated through a two-stage pipeline:
Transcript Generation: We used GPT-4o-mini to generate Dutch transcripts that match the word count distribution observed in Common Voice, ensuring realistic utterance lengths and diverse linguistic content.
Speech Synthesis: Each transcript was converted to audio using OpenAI's TTS-1 model with 9 different voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer), producing 34,898 synthetic samples.
Quality Filtering with WAVe: Raw synthetic speech often contains defects such as mispronunciations, omitted words, or prosodic anomalies. To address this, we applied WAVe (Word-Aligned Verification), a model that assesses audio-text alignment at the word level rather than the sentence level. WAVe uses multi-head attention to align each word to its corresponding audio frames and assigns per-word confidence scores via a GLU-based scorer. For this model, only samples scoring above the strict threshold (q ≥ 0.8) were retained, resulting in 10,555 high-quality synthetic samples.
How the Model Was Created
The model was fine-tuned from openai/whisper-large-v3 using the Hugging Face Transformers library with the following approach:
Mixed Training: Combined 34,952 real speech samples from Common Voice 17.0 Dutch with 10,555 strictly WAVe-filtered high-quality synthetic samples (45,507 total).
Optimization: Trained for 5 epochs with a learning rate of 5e-6, global batch size of 256, and BF16 precision on an NVIDIA H200 GPU.
Checkpoint Selection: The best checkpoint was selected based on validation loss, occurring at step 350 with a validation loss of 0.0552.
This high-quality filtering approach achieves 35% reduction in training steps compared to using all synthetic data, while maintaining excellent ASR performance.
Model Details
Property
Value
Base Model
openai/whisper-large-v3
Language
Dutch (nl)
Task
Automatic Speech Recognition (transcribe)
Parameters
1550M
Training Data
Common Voice 17.0 + High-Quality Synthetic (q ≥ 0.8)
Total Training Samples
45,507
Sampling Rate
16kHz
Evaluation Results
This Model (whisper-large-v3-high-mixed-nl)
Metric
Value
Validation Loss
0.0520
Validation WER
3.57%
Test WER (Common Voice)
4.43%
Test WER (MLS)
20.29%
Best Checkpoint
Step 350
Max Training Steps
890
Comparison with Other Training Configurations (Whisper-Large-v3 Dutch)
Training Data
Max Steps
Val Loss
Val WER
Test WER (CV)
Test WER (MLS)
Common Voice Only
680
0.0549
3.56%
4.39%
22.43%
High-Quality Filtered + CV
890
0.0520
3.57%
4.43%
20.29%
Mid-High Quality Filtered + CV
1,270
0.0570
3.63%
4.48%
17.25%
All Synthetic + CV (Unfiltered)
1,365
0.0560
3.61%
4.44%
17.02%
Key Performance Highlights
Most efficient training: Only 890 max steps (35% fewer than unfiltered)
Best validation loss (0.0520) among all Whisper-Large-v3 Dutch configurations
Competitive in-domain performance: 4.43% Test WER on Common Voice
9.5% relative improvement on MLS benchmark vs baseline (20.29% vs 22.43%)
Best quality-to-compute ratio: Strong results with only top-tier synthetic data (30.2%)
