This is a fine-tuned version of openai/whisper-small model on Vietnamese speech data. The model aims to improve transcription accuracy and robustness for Vietnamese automatic speech recognition (ASR) tasks, especially in real-world scenarios.

📊 Fine-tuning Results

Word Error Rate (WER): 9.3485

Evaluation was performed on a held-out test set with diverse regional accents and speaking styles.

📝 Model Description

The Whisper small model is a transformer-small sequence-to-sequence model designed for automatic speech recognition and translation tasks. It has been trained on over 680,000 hours of labeled audio data in multiple languages. The fine-tuned version of this model focuses on the Vietnamese language, aiming to improve transcription accuracy and handling of local dialects.

This model works with the WhisperProcessor to pre-process audio inputs into log-Mel spectrograms and decode them into text.

📁 Dataset

Total Duration: More 250 hours of high-quality Vietnamese speech data
Sources: Public Vietnamese datasets
Format: 16kHz WAV files with corresponding text transcripts
Preprocessing: Audio was normalized and segmented. Transcripts were cleaned and tokenized.

🚀 How to Use

To use the fine-tuned model, you can follow these steps:

Install the required dependencies:

python
1# Install required libraries 2!pip install transformers torch librosa soundfile --quiet 3 4# Import necessary libraries 5import torch 6import librosa 7import soundfile as sf 8from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline 9 10print("Environment setup completed!")
Use the model for inference:

python
1import torch 2import librosa 3from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq 4 5device = "cuda" if torch.cuda.is_available() else "cpu" 6print(f"Using device: {device}") 7 8# Load processor and model 9model_id = "namphungdn134/whisper-small-vi" 10print(f"Loading model from: {model_id}") 11processor = AutoProcessor.from_pretrained(model_id) 12model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).to(device) 13 14# config language and task 15forced_decoder_ids = processor.get_decoder_prompt_ids(language="vi", task="transcribe") 16model.config.forced_decoder_ids = forced_decoder_ids 17print(f"Forced decoder IDs for Vietnamese: {forced_decoder_ids}") 18 19# Preprocess 20audio_path = "example.wav" 21print(f"Loading audio from: {audio_path}") 22audio, sr = librosa.load(audio_path, sr=16000) 23input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device) 24print(f"Input features shape: {input_features.shape}") 25 26# Generate 27print("Generating transcription...") 28with torch.no_grad(): 29 predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids) 30 31transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] 32print("📝 Transcription:", transcription) 33 34# Debug: Print token to check 35print("Predicted IDs:", predicted_ids[0].tolist())

⚠️ Limitations

This model is specifically fine-tuned for the Vietnamese language. It might not perform well on other languages.
Struggles with overlapping speech or noisy background.
Performance may drop with strong dialectal variations not well represented in training data.

📄 License

This model is licensed under the MIT License.

📚 Citation

If you use this model in your research or application, please cite the original Whisper model and this fine-tuning work as follows:

@article{Whisper2021,
  title={Whisper: A Multilingual Speech Recognition Model},
  author={OpenAI},
  year={2021},
  journal={arXiv:2202.12064},
  url={https://arxiv.org/abs/2202.12064}
}

@misc{title={Whisper small Vi V1.1 - Nam Phung},
  author={Nam Phùng},
  organization={DUT},
  year={2025},
  url={https://huggingface.co/namphungdn134/whisper-small-vi},
  url={https://github.com/namphung134/ASR-Vietnamese}
}

📬 Contact

For questions, collaborations, or suggestions, feel free to reach out via [namphungdn134@gmail.com].