🧩 Robust ASR Model for Real-World Yoruba Speech (Domain Data Only)
This ASR model is trained exclusively on 50 hours of real-world, domain-specific Yoruba audio, including conversational and semi-spontaneous speech.
It is designed to handle noisy environments, diverse speaker styles, and more natural linguistic variation.
It does similarly well for clean and well-structured speech input
This model is part of a full ASR ablation study that analyzes and understands the robustness of data and in dealing with different modes and variations of data collections.
👉 View all models on
GitHubWe are particularly interested in validating the conclusions we’ve observed through our ablation studies:
While benchmark datasets like FLEURS are useful for comparison, they do not fully capture the variability and challenges of real-world speech — especially for underrepresented languages like Swahili and Yoruba.
We are inviting the community to try out these models and help assess:
- How well the models perform on natural, conversational, or noisy audio
- Open-source datasets (like Common Voice & FLEURS) perform well on clean, benchmark speech.
- Whether the improvements we've seen in combining diverse datasets generalize to your use case
- Gaps between benchmark results and real-world usability
- A combination of both yields balanced results but depends on data quality and label accuracy.
Model
Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.
🚀 How to Use
1from transformers import WhisperForConditionalGeneration, WhisperProcessor
2from transformers import pipeline
3from transformers.utils import is_flash_attn_2_available
4
5processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
6model = WhisperForConditionalGeneration.from_pretrained("RafatK/Whisper_Largev2-Yoruba-Decodis_FT", torch_dtype=torch.float16).to("cuda")
7model.generation_config.input_ids = model.generation_config.forced_decoder_ids
8model.generation_config.forced_decoder_ids = None
9
10pipe = pipeline(
11 "automatic-speech-recognition",
12 model=model,
13 processor = "openai/whisper-large-v2",
14 tokenizer = "openai/whisper-large-v2",
15 feature_extractor = "openai/whisper-large-v2",
16 chunk_length_s=15,
17 device=device,
18 model_kwargs={"attn_implementation": "flash_attention_2"} if is_flash_attn_2_available() else {"attn_implementation": "sdpa"},
19 generate_kwargs = {
20 'num_beams':5,
21 'max_new_tokens':440,
22 'early_stopping':True,
23 'language': 'english',
24 'task': 'transcribe'
25 }
26)
27text_output = pipe("audio.wav")['text']
📦 Training Data
- Custom real-world dataset
- Yoruba
- Collected from real use cases (e.g. mobile recordings, community sources)
- ~23 hours
- Not publicly released (due to licensing)
📁 Languages: Yoruba (yo)
🏋️♂️ Training Setup
- Architecture:
whisper-large-v2
- Framework: Whisper and Huggingface Transformers
- Sampling rate: 16 kHz
- Preprocessing: Volume normalization, High-Grade noise addition and filtering, Prosodic Augmentation,silence trimming
- Learning Rate: 1e-5
- Optimizer: Adamw_pytorch
- Steps: 3000
📈 Evaluation
| Dataset | This Model | Whisper Large V2 |
|---|
| FLEURS (benchmark) | 66.22 | No-Info |
| Our test set | 64.51 | No-Info |
🎯 Intended Use
This model is best for:
- Noisy, real-world speech input
- Community-contributed or semi-structured conversation
- Language tools for low-resource environments
⚠️ Limitations
- Underperforms on clean datasets like FLEURS mainly due to size of train set
- May exhibit bias toward some accents
- Limited by the smaller training size (23h)
📝 Please try the models and share your feedback, issues, or results via:
GitHub Issues: Submit an issue
Hugging Face Discussions: Join the conversation
Your feedback will help us refine our dataset and improve ASR for underrepresented languages like Swahili and Yoruba.
