This model is a fine-tuned version of openai/whisper-small on the MiZonal v3.0 dataset. Note: ~1 hour of conversational speech was added to this dataset version.

It achieves the following results on the evaluation set:

Wer: 22.3422
Cer: 8.0155
Real Time Factor: 0.0255

Quick Inference

python

import torch
import librosa
from transformers import WhisperProcessor, WhisperForConditionalGeneration

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = WhisperProcessor.from_pretrained("andrewbawitlung/whisper-small-mizonal3-E2-lus-v2026.06")
model = WhisperForConditionalGeneration.from_pretrained("andrewbawitlung/whisper-small-mizonal3-E2-lus-v2026.06").to(device)

audio, sr = librosa.load("your_audio.wav", sr=16000)
input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(input_features, max_new_tokens=256)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
print(transcription)

Model description

Experiment Configurations

This repository is part of a series of experiments. The different configurations are:

E1 (Baseline): Standard training configuration.
E2 (Noise): Training with background noise augmentation.
E3 (Speed): Training with speed perturbation augmentation.
E4 (SpecAug): Training with SpecAugment (time and frequency masking).
E5 (Combined): Training with a combination of all augmentations.

All Models in this Family

Experiment	Hugging Face Repository
E1 (Baseline)	andrewbawitlung/whisper-small-mizonal3-E1-lus-v2026.06
E2 (Noise)	andrewbawitlung/whisper-small-mizonal3-E2-lus-v2026.06
E3 (Speed)	andrewbawitlung/whisper-small-mizonal3-E3-lus-v2026.06
E4 (SpecAug)	andrewbawitlung/whisper-small-mizonal3-E4-lus-v2026.06
E5 (Combined)	andrewbawitlung/whisper-small-mizonal3-E5-lus-v2026.06

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0003
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: OptimizerNames.ADAMW_TORCH_FUSED
lr_scheduler_type: SchedulerType.LINEAR
num_epochs: 8

Training results

step	epoch	train_loss	eval_loss	eval_wer	eval_cer	learning_rate	grad_norm
250	0.46	2.4902	0.5650	37.47	12.96	1.49e-04	29.28
500	0.91	2.6669	0.7054	39.37	16.05	2.99e-04	27.87
750	1.37	2.0718	0.6896	38.85	25.33	2.81e-04	24.14
1000	1.82	1.5676	0.6065	31.95	11.21	2.62e-04	15.83
1250	2.28	1.0195	0.6181	30.51	12.53	2.42e-04	20.85
1500	2.73	0.8523	0.5772	30.07	12.88	2.23e-04	11.63
1750	3.19	0.5502	0.5918	28.73	11.14	2.04e-04	7.99
2000	3.64	0.5025	0.5752	30.36	11.42	1.84e-04	12.55
2250	4.10	0.3019	0.5884	33.95	15.83	1.65e-04	5.13
2500	4.55	0.2702	0.5884	40.83	23.26	1.46e-04	4.48
2750	5.01	0.2087	0.5710	23.39	8.77	1.27e-04	5.55
3000	5.46	0.1572	0.5716	25.11	10.18	1.07e-04	6.26
3250	5.92	0.1184	0.5987	32.94	15.67	8.81e-05	2.89
3500	6.38	0.0607	0.5804	21.96	7.76	6.88e-05	0.80
3750	6.83	0.0562	0.5838	21.34	7.26	4.96e-05	9.40
4000	7.29	0.0172	0.5829	21.33	7.35	3.03e-05	0.17
4250	7.74	0.0087	0.5896	22.58	8.39	1.10e-05	0.07
4392	8.00		0.5884	21.92	8.03