This model is a fine-tuned version of openai/whisper-small trained on juanjucm/OpenHQ-SpeechT-GL-EN dataset for Galician-to-English Text to Speech Translation task. It takes galician speech audios as input and generates the correspondant translated transcription in English.

The motivation behind this work is to increase the visibility of the Galician language, making it more accessible for non-Galician speakers to understand and engage with Galician audio content.

This model was developed during a 3-week Speech Translation workshop organised by Yasmin Moslem.

Performance and training details

Baseline model achieved a BLEU score of 22.8 on the evaluation dataset.

After fine-tuning, it achieves the following results on the evaluation set:

Loss: 0.7134
Wer: 37.6147
BLEU: 50.9676
ChrF++: 69.24

The following hyperparameters were used during training:

learning_rate: 1.25e-05
train_batch_size: 16
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 2
total_train_batch_size: 32
total_eval_batch_size: 16
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 5
mixed_precision_training: Native AMP

Training results

We used BLEU Score as our reference translation metric for selecting the best checkpoint after training.

Training Loss	Epoch	Step	Validation Loss	Wer	Bleu
0.3055	1.0	150	0.6858	40.7110	47.7920
0.0378	3.0	450	0.6819	38.6959	49.8621
0.013	4.0	600	0.7134	37.6147	50.9676
0.014	5.0	750	0.7308	38.0570	49.2006

Framework versions

Transformers 4.45.1
Pytorch 2.4.1+cu121
Datasets 3.0.1
Tokenizers 0.20.0

juanjucm/whisper-small-OpenHQ-GL-EN

whisper-small-OpenHQ-GL-EN

Performance and training details

Training results

Framework versions

juanjucm/
whisper-small-OpenHQ-GL-EN