It achieves the following WER results on the evaluation set:
Normalised WER: 6.324
Orthographic WER: 8.233
Full tensorboard logs can be found under the tab Training Metrics,
and steps to reproduce here.
Model description
We copy the entire encoder module and freeze it during training. We copy only two decoder layers, which are initialised from the first and last decoder layers from Whisper. All other decoder layers from Whisper are discarded.
The model is trained on a knowledge distillation objective. Specifically, it is trained to minimise the KL divergence between the distilled model and the Whisper model, as well as the cross-entropy loss on the labelled Common Voice audio data.
For more details, refer to the Distil-Whisper repository and paper.
Training and evaluation data
The model was trained and evaluated on the German subset of the Common Voice 15.0 dataset.
Training procedure
To reproduce this training run, first clone and install Distil-Whisper according to the instructions here.
Next, we can pick a name for our distilled model, e.g. distil-whisper-large-v3-de-kd. We can then run the following command to create a repository under this name:
The following command demonstrates how to initialise a student model from the Whisper large-v3
checkpoint, with all 32 encoder layer and 2 decoder layers. The 2 student decoder layers are copied from teacher layers
1 and 32 respectively, as the maximally spaced layers:
The initialised model will be saved to the sub-directory distil-large-v3-init in our model repository, ready to be trained.
We can then train the model for a total of 50k steps on the German subset of the Common Voice 15 dataset by executing the following command. Note that we train
directly on the text labels provided in the Common Voice dataset, rather than first pseudo-labelling the dataset as was done in the original Distil-Whisper paper:
On a single 80GB A100 GPU, training will take approximately 3.5 days (or 85 hours), and reach a final WER of 6.3%. Tensorboard logs can be found under the tab Training Metrics.
Note that training for longer would likely have improved the final WER performance further, since the model had not fully converged after 50k train steps.
Training hyperparameters
The following hyperparameters were used during training:
learning_rate: 1e-04
train_batch_size: 64
eval_batch_size: 64
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
