Fine-tuned openai/whisper-small on approximately 10 hours of conversational Hindi speech, achieving a WER reduction from 48.26% → 31.51% on the in-domain evaluation set.
Model Description
This model is a domain-adapted version of Whisper-small, fine-tuned for conversational Hindi ASR. The base model was adapted using a carefully engineered utterance-level dataset derived from 104 long-form Hindi recordings. Key contributions include a custom data pipeline, disfluency-aware dataset construction, and a novel lattice-based multi-system consensus evaluation framework.
Model type: Whisper (encoder-decoder Transformer)
Language: Hindi (hi)
Task: Automatic Speech Recognition (ASR)
Base model: openai/whisper-small (244M parameters)
Fine-tuned by: Joshuva Vinith
License: MIT
Training Data
The training corpus was constructed from scratch from 104 long-form conversational Hindi recordings (~10 hours total). The pipeline involved:
Programmatic URL reconstruction and audio retrieval
JSON-aligned utterance-level segmentation using pydub
Audio standardization to 16 kHz mono using librosa
Removal of 209 redacted segments for privacy
Final usable set: 5,732 utterance clips
A unique-word spelling analysis across 7,457 transcript tokens found 65.5% were orthographically noisy, directly informing text normalization strategy.
Training Configuration
Parameter
Value
Base model
whisper-small
Epochs
4 (best at epoch 3)
Batch size
8
Learning rate
1e-5
Optimizer
AdamW
Weight decay
0.01
Warmup steps
500
Mixed precision
FP16
Gradient checkpointing
Enabled
Framework
HuggingFace Seq2SeqTrainer
Evaluation Results
In-domain (Conversational Hindi)
Epoch
Validation Loss
WER (%)
1
0.3356
39.24
2
0.3030
33.01
3
0.3120
31.51
4
0.3606
32.48
Baseline (pretrained Whisper-small): 48.26% WER Best fine-tuned (epoch 3): 31.51% WER Improvement: 16.75 percentage points (~35% relative reduction)
Cross-domain Generalization
The fine-tuned model was also evaluated on an external clean read-speech Hindi benchmark. It consistently outperformed the pretrained Whisper-small baseline on this out-of-domain set, demonstrating that fine-tuning improved general Hindi ASR capability beyond the conversational domain.
Lattice-Based Consensus Evaluation
A novel lattice-style multi-system consensus framework was designed to evaluate 6 ASR hypotheses jointly via Levenshtein dynamic programming with gap modeling. Most models showed reduced WER under the lattice reference vs a single human transcript, indicating that some apparent errors were reference disagreements rather than true recognition mistakes.
