A Russian-specialized openai/whisper-large-v3-turbo fine-tune with English preserved. Stage AS delivers punctuation and capitalization restoration on top of a strong Russian WER. v2 adds a rare-mark-weighted training loop and 3-epoch schedule that dropped aggregate WER by nearly a full point vs v1.

Headline metrics (6 RU test sets, N=500 each)

Metric	Vanilla turbo	Stage AI (prev)	Stage AS v1	Stage AS v2 (this)
WER (aggregate)	13.25 %	9.43 %	9.83 %	8.93 %
PER F1 (punct macro F1)	≈ 0.05	0.20	0.39	0.41
Cap F1 (word-level case-match)	≈ 0.20	0.48	0.83	0.83
Sent F1 (sentence-boundary F1)	≈ 0.15	0.45	0.81	0.80

v2 vs v1: WER −0.90 pp, PER +0.02, Cap ~stable, Sent ~stable.

Metric correction (2026-07-19). The per-mark F1 table below was regenerated after fixing a bug in the evaluation harness that ignored leading punctuation on tokens (Russian dialogue em-dash — слово… and standalone ...). The trained model was actually emitting these marks all along; only the metric didn't see them. Aggregate WER / Cap / Sent are unaffected. The correct em-dash F1 for v2 is 0.322, not the 0.000 previously reported.

Per-dataset delta (Stage AS v1 → v2)

Dataset	WER%	PER F1	Cap F1	Sent F1
Common Voice 21 RU	−0.5	+0.00	+0.01	+0.01
RuLibriSpeech	−0.9	+0.02	+0.03	+0.02
Sberdevices Golos far-field	−2.0	+0.05	+0.01	+0.01
Sberdevices Golos crowd	−1.9	+0.05	+0.01	+0.03
SOVA RuDevices	+0.1	+0.01	+0.02	−0.01
Podlodka Speech	−0.3	−0.01	−0.05	−0.11

Per-mark F1 (v2, corrected metric)

Mark	v2	non-zero on datasets
. period	0.759	6 / 6
, comma	0.734	6 / 6
? question	0.770	6 / 6
! exclamation	0.381	5 / 6
— dash	0.322	6 / 6
… ellipsis	0.038	1 / 6
« quote_open	0.176	3 / 6
» quote_close	0.159	4 / 6

Dash performance is strongest on Common Voice (F1 0.612) and degrades on the noisier Golos / SOVA sources. Ellipsis remains hard to measure — only SOVA test refs contain gold ellipsis, and only 29 rows across ~2 400 aligned test samples. quote_open/quote_close low scores are chiefly a reference-set artifact: quotes were only present in the augmented SOVA and Golos refs and inconsistently formatted.

Real-world qualitative check: dialogue lines from Russian audiobooks and podcasts produce dashes on speaker turns; short quoted phrases in news / interview audio produce guillemets. The metric here understates what the model actually does on well-formatted references.

What was fixed

The previous Stage AI release used GigaAM pseudo-labels for the hardtail samples in training. GigaAM emits mostly periods and sentence-initial capitals — Stage AI inherited that poverty. Around 46 % of the training-label corpus had zero punctuation by construction.

Stage AS v1: re-labelled the punct-poor slice via an offline LLM pass (Gemma 3 12B, WER ≤ 1 % word-preservation verification), yielding a 108 K-row publication-quality training corpus. LoRA fine-tune (r=32, q/k/v/o + fc1/fc2, 2 epochs, lr 5e-6).

Stage AS v2 (this release): larger LoRA (r=48, 42 M trainable params), 3 epochs, WeightedRandomSampler ×2 on rows containing rare marks (—, –, …, !, ?) so 32 % of samples surface at double frequency. WER dropped nearly a point aggregate — including −2 pp on both Golos splits.

Evaluation methodology

PER macro F1 — per-mark F1 (period, comma, question, exclamation, em-dash, ellipsis, quotes) averaged across mark types that appear at least once in the reference.
Cap F1 — per-word case-match F1 over words that survive ref↔hyp alignment.
Sent F1 — sentence-boundary F1 based on [.!?…] positions, 0.02-fraction tolerance for matching.
WER — standard, computed on lowercase punct-stripped text.

Golos and SOVA test-set references lack punctuation — measuring punct/cap F1 against them without augmentation always yields 0 for both models. The aggregate row above uses augmented references from Gemma 3 12B with the same word-preservation verification as training (1256 augmented / 234 rejected / 0 errors over Golos+SOVA). Both v1 and v2 evaluated on the same augmented references. Un-augmented CV21 / RuLibri / Podlodka refs are kept as-is.

Usage

python

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

processor = WhisperProcessor.from_pretrained("coriollon/whisper-large-v3-turbo-russian")
model = WhisperForConditionalGeneration.from_pretrained(
    "coriollon/whisper-large-v3-turbo-russian",
    torch_dtype=torch.float16,
).to("cuda")

inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to("cuda", dtype=torch.float16)
predicted_ids = model.generate(input_features, language="ru", task="transcribe")
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

Pre-quantized faster-whisper variants

Folder	Quantization	Size
`ct2_int8_float16/`	int8 weights + fp16 compute	782 MB
`ct2-int16/`	int16 (Stage AS v1 build; rebuild pending)	1.6 GB

python

from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel

ct2_path = snapshot_download(
    repo_id="coriollon/whisper-large-v3-turbo-russian",
    allow_patterns="ct2_int8_float16/*",
)
model = WhisperModel(f"{ct2_path}/ct2_int8_float16",
                    device="cuda", compute_type="int8_float16")
segments, _ = model.transcribe("audio.wav", language="ru", beam_size=5)

Limitations

English-only audio degraded vs vanilla turbo (Russian specialization tradeoff — inherited from Stage AI).
For pure-Russian deployments with no punctuation or case requirement, dedicated Russian ASR models like GigaAM outperform this on raw WER.
Ellipsis (…) F1 stays low: only SOVA test refs contain any gold ellipsis; the measured 0.038 is not a confident estimate.
On the un-augmented CV21 / RuLibri / Podlodka references the punct gains over vanilla turbo are smaller than the augmented-refs aggregate suggests; they're still positive.

License

Apache 2.0 (inherited from base Whisper model).

Changelog

Metric fix (2026-07-19) — Evaluation harness now counts leading em-dash (Russian dialogue) and standalone ... / … tokens. No change to the released weights, only to how their punctuation quality is reported. Corrected numbers: WER 8.93 %, PER 0.41, dash F1 0.322 (was reported as 0.000).
Stage AS v2 (2026-07-14) — LoRA r=48, 3 epochs, rare-mark ×2 sample weighting. WER 9.83 → 8.93 %. PER 0.39 → 0.41, Cap and Sent essentially preserved. Golos WER drops −2 pp on both splits.
Stage AS v1 (2026-07-12) — punctuation & capitalization LoRA fine-tune. Cap F1 0.48 → 0.83, Sent F1 0.45 → 0.81, PER F1 0.20 → 0.39. WER 9.43 → 9.83 %.
Stage AI (previous) — GigaAM-teacher relabel, WER 13.25 → 9.43 %.