Model Card for whisper-medium-ht
Model Details
Model Description
This model is a fine-tuned whisper medium model using general conference talks from the Church of Jesus Christ of Latter-day Saints.
- Developed by: Zachary Clement
- Model type: Automatic Speech Recognition
- Language(s): Haitian Creole
- License: Apache 2.0
- Finetuned from model [optional]: openai/whisper-medium
Model Sources [optional]
- Repository: See the fine_tuning directory in my Github. Note that this was the phonetic only candidate.
Uses
This model should be used to convert Haitian Creole audio to text. I did not check for catastrophic forgetting and do not recommend using on other languages.
Bias, Risks, and Limitations
There is a risk of catastrophic forgetting for languages other than English
Training Details
Training Data
This model was trained using Haitian Creole transcriptions of
General Conference meetings for the Church of Jesus Christ of Latter-day Saints.
Audio from general conference talks were broken into segments, ASR was used to get a garbled transcription of audio, and an LLM was used to match the garbled ASR outputs onto the transcriptions.
In total, 9,838 training samples were used, comprising 41 hours of labeled data.
Training Procedure
Preprocessing
Audio is resampled to 16 kHz mono. Each sample is transformed into an 80-bin log-mel spectrogram using Whisper's WhisperFeatureExtractor (30-second context window).
Transcriptions are tokenized with WhisperTokenizer configured for Haitian Creole (ht) transcription. Samples whose tokenized label exceeds 448 tokens are dropped before
training. No audio augmentation is applied to the phonetic candidate — it trains on original phonetic-alignment segments only.
The dataset is split by talk ID: 15 specific talks are held out as a fixed evaluation set and fully excluded from all training splits (no segment from an eval talk appears in
any training candidate, including synthetic).
Training Hyperparameters
- Training regime: bf16 mixed precision
- Base model: openai/whisper-medium
- Task/language: transcribe / Haitian Creole (ht)
- Encoder: fully trainable (no freezing)
- Max steps: 3,000
- Per-device train batch size: 16
- Gradient accumulation steps: 2 → effective batch size: 32
- Per-device eval batch size: 8
- Learning rate: 1e-5 with cosine decay
- Warmup steps: 500
- Gradient checkpointing: enabled
- Eval/save strategy: every 500 steps (steps // 6), best checkpoint retained by WER
- Save total limit: 3 checkpoints
- Generation max length: 225 tokens
- Optimizer: AdamW (HuggingFace Trainer default)
Speeds, Sizes, Times [optional]
- Hardware: single NVIDIA L4 (24 GB VRAM), 16 GB RAM, 4 vCPUs
- Training duration: 3,000 steps
- Logging: every 25 steps (TensorBoard)
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
Testing data: 15 held-out Haitian Creole conference talks, excluded from all training splits. Talk IDs: 2012_10_Nourise_a, 2021_10_Diyite_Pa_Vledi_Nou_Pafe,
2021_4_Kisa_Sove_nou_an_te_fe_pou_nou, 2021_10_Lanmou_Bondye_Sa_ki_bay_nanm_lan_plis_lajwa, 2012_10_Bon_Dezi_pou_Angaje, 2012_10_Eprev_lafwa_nou,
2015_4_Prezidan_Dieter_F_Uchtdorf, 2019_10_Bay_Espri_nou_Kontwol_Sou_Ko_nou, 2021_4_Chemen_alyans_lan, 2019_4_Jan_l_te_fe_a, 2014_4_Fotifye_nou_e_pran_kouraj,
2021_4_Bondye_Nan_Mitan_Nou, 2018_10_Vin_tounen_Sen_Denye_Jou_Egzanple, 2015_10_Elde_Quentin_L_Cook, 2012_4_Rapo_Depatman_Odit_Legliz_la_2011.
Metrics
- WER (Word Error Rate) — primary metric; used for best-checkpoint selection during training
- CER (Character Error Rate) — secondary metric
Results
The fine-tuned model showed substantial improvements over other whisper models on the holdout set.
| Model | WER | CER |
|---|
| openai/whisper-large-v3 | 69.13% | 31.30% |
| openai/whisper-medium | 84.25% | 40.89% |
| openai/whisper-small | 97.28% | 48.84% |
| clementzach/whisper-medium-ht | 34.13% | 20.30% |
