All the modelling work is ivrit.ai's. This repository only adds a Q4_K
quantization of their published weights, so the model fits on machines where
the 1.5 GB original does not, and measures what that costs on Hebrew.
Why this exists: stock Whisper treats Hebrew as one of 100 languages and
does it poorly. On FLEURS Hebrew, stock Large v3 Turbo scores 28.59% WER.
This file is 3.4× smaller than the stock model and scores 14.79% —
roughly half the errors.
Quantization cost, relative to the unquantized f16 upstream weights (1549 MB,
14.37% WER): +0.42 percentage points for 3.4× less disk.
If you want the last 0.42 points, quantize the upstream f16 yourself with
whisper.cpp's whisper-quantize — Q5_0 measured identical to f16 here
(14.37%) at 547 MB.
Important: set the language to Hebrew explicitly
The upstream model card warns that language detection was degraded during
fine-tuning:
Language detection capability of this model has been degraded during
training — it is intended for mostly-hebrew audio transcription. Language
token should be explicitly set to Hebrew.
Translation was likewise not trained. Always pass -l he (or select
Hebrew in your app). Auto-detect will cost you accuracy. Every number here was
measured with the language pinned to Hebrew.
Usage
whisper.cpp
./whisper-cli -m ivrit-ai-whisper-large-v3-turbo-q4_k.bin -l he -f audio.wav
Handy
Handy is a free, open-source, offline speech-to-text
app for Windows, macOS and Linux. It picks up GGML .bin files dropped into
its models directory:
Copy the .bin there, restart Handy (or press Rescan on the Models screen),
then pick the model and set the language to Hebrew.
Note: legacy .bin files carry no capability metadata, so Handy cannot know
the model's language and it will not appear under a Hebrew language filter.
It works fine once selected.
Also works with Vibe and anything else
built on whisper.cpp.
40 utterances, 6.4 minutes of audio, sampled deterministically across the
347 unique sentences in the split
whisper.cpp b4938, CPU + BLAS
Greedy decoding (-bs 1 -bo 1), language pinned (-l he)
Identical flags for every model; only -m differs
WER: Levenshtein over whitespace-separated tokens, punctuation stripped,
Hebrew niqqud removed, geresh folded to ASCII apostrophe
Model
Size
corpus WER
mean
median
Stock whisper-large-v3-turbo
1549 MB
28.59%
27.91%
22.65%
ivrit.ai f16 (upstream)
1549 MB
14.37%
13.87%
9.17%
ivrit.ai Q5_0
547 MB
14.37%
13.92%
8.01%
ivrit.ai Q4_K (this file)
452 MB
14.79%
14.57%
8.01%
Per-utterance, the fine-tune beat stock on 32 of 40 clips, tied on 7, lost
on 1. A paired bootstrap over 10,000 resamples puts this file's mean-WER gain
over stock at +13.4 points (95% CI 9.0–19.0).
Median WER is identical to Q5_0 (8.01%) — on a typical clip there is no
difference; the gap only shows on harder ones.
Limitations of these numbers
Stated plainly, because they are easy to over-read:
All speakers are male. FLEURS he_il test and dev splits contain only
male speakers (792/792). Female speakers exist only in train, which was
excluded to avoid contamination. The comparison is controlled — every model
heard identical audio — but these numbers say nothing about female speech.
Greedy decoding, not beam search. This inflates absolute WER for every
model equally; relative comparisons are unaffected. Beam search would lower
all figures.
n = 40, not the full 792-utterance split. The confidence interval
accounts for this.
Q5_0 and f16 both scored 14.37%. That is a coincidence of integer error
counts at this sample size, not identical output — their transcripts differ
on 8 of 40 clips.
Credits
ivrit.ai — the Hebrew fine-tune and all the
training work, released under Apache-2.0. Trained on crowd-transcribe-v5,
crowd-recital-whisper-training and knesset-plenums-whisper-training.
