This model was developed to improve Whisper's performance on conversational Hindi and code-mixed Hindi-English speech using a carefully curated synthetic speech corpus.

Model Details

Property	Value
Base Model	openai/whisper-medium
Model Type	Whisper Encoder-Decoder Transformer
Task	Automatic Speech Recognition
Languages	Hindi, Hinglish
Framework	Hugging Face Transformers

Motivation

OpenAI Whisper provides excellent multilingual speech recognition, but conversational Hindi and Hinglish often remain challenging due to code-mixing, pronunciation variations, and domain-specific vocabulary.

This project fine-tunes Whisper Medium using a curated synthetic dataset to significantly improve recognition quality while preserving Whisper's multilingual capabilities.

Dataset

The training dataset was generated using a multi-stage synthetic data generation pipeline consisting of:

LLM-generated conversational Hindi/Hinglish text
High-quality speech synthesis
Transcript refinement and normalization
Quality filtering
Curated speech-text pairs for Whisper fine-tuning

The focus was on maximizing transcript quality while maintaining linguistic diversity.

Evaluation

Evaluation was performed on an unseen held-out test set.

Overall Performance

Metric	Score
Test Samples	373
Word Error Rate (WER)	0.0456
Character Error Rate (CER)	0.0262
Relative WER Improvement	86.99%

Performance by Difficulty

Difficulty	Samples	WER
Easy	317	0.0303
Medium	49	0.1301
Hard	7	0.2182

Performance by Language

Language	Samples	WER
Hindi	280	0.0514
Hinglish	93	0.0300

Performance by Error Category

Category	Samples	WER
Perfect	288	0.0384
Minor Phonetic	50	0.0628
Digit Format	29	0.0758
Script Switch	3	0.0682
Genuine Error	2	0.0588
Loanword Mishear	1	0.0000

Usage

Load the model using the Hugging Face Transformers library.

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

processor = WhisperProcessor.from_pretrained(
    "shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)

model = WhisperForConditionalGeneration.from_pretrained(
    "shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned"
)

device = "cuda" if torch.cuda.is_available() else "cpu"

model = model.to(device)
model.eval()

Intended Uses

This model is intended for:

Hindi Automatic Speech Recognition
Hinglish Automatic Speech Recognition
Academic Research
Speech Recognition Experiments
Fine-tuning Research

Limitations

The model is primarily trained on synthetic speech.
Performance may decrease on noisy recordings.
Performance may vary across unseen accents and speaking styles.
Not evaluated for streaming ASR.

Training Framework

Hugging Face Transformers
PyTorch

Results Summary

Metric	Value
WER	0.0456
CER	0.0262
Test Samples	373
WER Improvement	86.99%

Acknowledgements

This project builds upon:

OpenAI Whisper
Hugging Face Transformers

Citation

If you use this model in your research or applications, please cite this Hugging Face repository.

bibtex

@misc{whisper_medium_hindi_hinglish_asr,
  title={Whisper Medium Hindi-Hinglish ASR (Fine-Tuned)},
  author={Suja Akhter},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/shujaAK/whisper-medium-hindi-hinglish-asr-fine-tuned}}
}