Language detection capability of this model has been degraded during training - it is intended for mostly-hebrew audio transcription. Language token should be explicitly set to Hebrew.

Additionally, the tanslation task was not trained and also degraded. This model would not be able to translate in any reasonable capacity.

How to Get Started with the Model

Please follow the original model card for usage details - replacing with this model name. You can also fine other weight formats ad quantizations on the ivrit ai HF page.

We created some simple example scripts using this model and weights for other inference runtimes. Find those in the "examples" folder within the training GitHub repo.

Training Details

Training Data

This model was trained on the following datasets:

ivrit-ai/crowd-transcribe-v5 - Publicly accessible audio sources have been crowd-transcribed segment-by-segment - ~300h
ivrit-ai/crowd-recital-whisper-training - Crowd-sourced recording of Wikipedia article snippets. ~50h
ivrit-ai/knesset-plenums-whisper-training - A subset of a Knesset (Israeli house of representatives) plenum protocols. ~4700h

Training Procedure

This model was trained in two main phases:

Knesset based pre-training - over all ~4700h of data - 3 epochs, ~54h run
Mixed post-training over all crowd-transcribe-v5 (300h), crowd-recital-whisper-training (50h) and highest-quality filtered knessets data (150h) - 1 epoch
Interleaving of datasets with sampling probs: (0.9, 0.025, 0.075) respectively
Note that crowd-transcribe-v5 has about 5x shorter samples on average thus the over-sampling.

This model is a weighted-average of the 3 lowest eval loss checkpoints from the same training run. Training code can be found on the ivrit-ai Github here

Preprocessing

The "Crowd Recital" and "Knesset" datasets contain timestamps and previous text following the Whisper expected inputs. Timestamps were used from 40% of samples from those datasets, and 50% of the previous text was used.

The "Crowd Transcribe" datasets has no timestamps or previous text and this preprocessing only included melspec feature extraction and text encoding.

Preprocessing code can be found within the training code repository.

Datasets were interleaved with 0.15:0.8:0.05 ratio (knesset:crowd-transcribe:crowd-recital).

Training Hyperparameters

Training regime: bf16 mixed precision with sdpa
Learning Rate: 1e-5, Linear decay, 800 steps warmup for 3 epochs
Batch Size: 32

Training Hardware / Duration

GPU Type: 8 x Nvidia A40 machine
Duration: ~60 run across both phases

Evaluation

Please refer to the ivrit-ai/hebrew-transcription-leaderboard

ivrit-ai/whisper-large-v3

Model Card for Model ID

Model Details

Model Description

Bias, Risks, and Limitations