Whisper Large v3 Turbo - French V6 (Anti-Overfitting)
Fine-tuned version of openai/whisper-large-v3-turbo for French ASR using aggressive anti-overfitting strategies .
🎯 Performance Overview
Average WER: 7.84% → 6.83% (-1.01pp, -12.9% improvement )
Evaluated with official Open ASR Leaderboard normalizer on 5 French datasets.
📊 Detailed Benchmark Results
Dataset Base Model Fine-tuned V6 Improvement Status VoxPopuli 12.11% 8.88% -3.23pp (-27%) 🚀 Excellent CommonVoice 7.57% 5.12% -2.45pp (-32%) 🚀 Excellent TEDx 8.52% 8.58% -0.06pp (-0.7%) ≈ Neutral MLS 5.61% 5.58% +0.03pp (+0.5%) ≈ Neutral FLEURS 5.41% 6.01% -0.60pp (-11%) ⚠️ Regression
image Key Findings
✅ Strong improvements on VoxPopuli (-27%) and CommonVoice (-32%)FLEURS regression - Model optimized for European French accents, not African accentsTEDx and MLS neutral (~0% change)3,176 samples evaluated across 5 datasets
🛡️ Anti-Overfitting Strategy V6
This version addresses overfitting issues observed in V5 through multiple strategies:
Reduced Model Capacity
LoRA Rank: 16 (vs 64 in V5, -75%)Target Modules: 6 (q_proj, k_proj, v_proj, out_proj, fc1, fc2)Trainable params: ~14M (vs 34M in V5, -59%)
Strong Regularization
Dropout: 0.2 (vs 0.05 in V5, +300%)Weight Decay: 0.05 (vs 0.01 in V5, +400%)Label Smoothing: 0.15 (vs 0.05 in V5, +200%)
Conservative Training
Learning Rate: 3e-5 (vs 1e-4 in V5, -70%)Epochs: 2 (vs 4 in V5, -50%)Training Samples: 97k (vs 122k in V5, -19%)
Strict Monitoring
Eval frequency: every 200 steps Early stopping patience: 2 (vs 3 in V5)
📊 Training Details
Base model: openai/whisper-large-v3-turboMethod: LoRA (r=16, alpha=32)Training samples: 97,193Validation samples: 2,000Epochs: 2Batch size: 64 (16 per device × 4 grad accum)Learning rate: 3e-05Weight decay: 0.05Label smoothing: 0.15Training time: 4.0h on A100 80GBNormalizer: Open ASR Leaderboard official
💾 Datasets (Reduced for Anti-Overfitting)
Carefully selected subset focusing on generalization:
Common Voice 23.0: 18k samples (-50%) - Reduced to prevent CV overfittingMultilingual LibriSpeech: 30k samples - Best generalizerVoxPopuli: 30k samples - Matches test distributionTEDx French: 12k samples (-20%) - Quality over quantityAfrican Accented French: 6k samples (-25%) - Diversity focusMediaSpeech: 3.2k samples - Full retention (small dataset)
Total: 97k samples (vs 122k in V5, -19%)
🚀 Usage
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from peft import PeftModel
import torch
Load model
processor = WhisperProcessor.from_pretrained("Mathos34400/whisper-large-v3-turbo-french-v6")
base_model = WhisperForConditionalGeneration.from_pretrained(
"openai/whisper-large-v3-turbo",
torch_dtype=torch.float16,
device_map="auto"
)
model = PeftModel.from_pretrained(base_model, "Mathos34400/whisper-large-v3-turbo-french-v6")
Inference
audio = ... # 16kHz numpy array
inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to("cuda")
generated_ids = model.generate(inputs["input_features"])
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)
🔬 V5 → V6 Changes
Problems in V5
Overfitting on FLEURS (6.25% WER)
Overfitting on VoxPopuli (13.24% WER)
Too much capacity (34M trainable params)
Average WER: 8.33%
V6 Solutions & Results
Capacity reduction: Rank 64 → 16 (-75%)Regularization: Dropout 0.05 → 0.2, Weight decay 0.01 → 0.05Less data: 122k → 97k samples (-19%)Shorter training: 4 → 2 epochsFrequent eval: Every 200 steps
Result: Average WER improved from 8.33% → 6.83% (-1.50pp, -18%)
🎯 Use Cases
✅ Recommended For:
Political speeches & debates (VoxPopuli-like)
General conversational French (CommonVoice)
Presentations & conferences (TEDx)
European French accents
Production deployments requiring speed & accuracy
⚠️ Not Optimal For:
African French accents (FLEURS)
Ultra-low WER requirements on audiobook-quality audio
📈 Comparison with V5
Metric V5 V6 Change VoxPopuli 13.24% 8.88% ✅ -4.36pp CommonVoice N/A 5.12% ✅ New FLEURS 6.25% 6.01% ✅ -0.24pp MLS 5.50% 5.58% ≈ +0.08pp Average 8.33% 6.83% ✅ -1.50pp
📝 Citation
@misc{whisper-french-v6,
author = {Mathis Lacombe},
title = {Whisper Large v3 Turbo French V6 - Anti-Overfitting},
year = {2025},
publisher = {Hugging Face},
howpublished = {
https://huggingface.co/Mathos34400/whisper-large-v3-turbo-french-v6}
}
📄 License
Apache 2.0
Model Card Authors: Mathis LacombeEvaluation Date: December 2025Evaluation Method: Official Open ASR Leaderboard normalizer
