Status: experimental Whisper adapter baseline. This adapter is kept for reproducibility and comparison. In the current frozen local evaluation, zero-shot facebook/mms-1b-all with rmc-script_latin performs better on both evaluated splits, so this repository should not be read as the strongest current Romani ASR result.

This repository contains a PEFT LoRA adapter for openai/whisper-large-v3-turbo, fine-tuned for Latin-script Romani ASR.

This is not a standalone Whisper checkpoint. Load the base model from Hugging Face, then load this adapter and the tokenizer/processor files from this repository.

This package is a slim adapter export. It omits redundant full base embedding/projection matrices and keeps the LoRA weights plus the trained <|romani|> token deltas. Before loading the adapter, initialize the custom <|romani|> token from Whisper's Slovak token (<|sk|>), then force the Romani decoder prompt.

The training/evaluation harness that produced this adapter contains the required loader path:

bash

python scripts/evaluate_hf_asr.py \
  --model-id openai/whisper-large-v3-turbo \
  --adapter-dir kiviki/whisper-turbo-romani-lora \
  --processor-dir kiviki/whisper-turbo-romani-lora \
  --language slovak \
  --task transcribe \
  --new-language-token romani \
  --max-length 72 \
  --no-repeat-ngram-size 3 \
  --repetition-penalty 1.15

Model Details

Base model: openai/whisper-large-v3-turbo
Adapter type: PEFT LoRA
LoRA rank: 16
LoRA alpha: 32
LoRA dropout: 0.05
Target modules: q_proj, v_proj
Encoder: frozen during training
Custom Whisper language token: <|romani|>
Decoder prompt: <|romani|><|transcribe|><|notimestamps|>

Frozen Evaluation

All results below were rerun on 2026-08-10 with fixed manifests and guarded decoding. Lower WER/CER is better.

Naming used below:

Prompt Test = test.csv, the original single-source prompt set (R_prompts).
Multi-Recording Dev = validation_clean.csv, the cleaned multi-source validation split.
Whisper Turbo Baseline = base Whisper with a Slovak transcription prompt and no Romani fine-tuning.
Whisper Turbo Romani LoRA = this fine-tuned adapter.
MMS-1B-All Romani/Latin Zero-Shot = public facebook/mms-1b-all with target_lang=rmc-script_latin, with no local MMS training.

Split	System	Role	WER	CER	ASCII WER	ASCII CER
Prompt Test	Whisper Turbo Baseline	unadapted baseline	0.796	0.184	0.719	0.138
Prompt Test	Whisper Turbo Romani LoRA	trained Whisper adapter	0.653	0.122	0.606	0.101
Prompt Test	MMS-1B-All Romani/Latin Zero-Shot	external zero-shot comparison	0.300	0.047	0.293	0.046
Multi-Recording Dev	Whisper Turbo Baseline	unadapted baseline	0.921	0.496	0.882	0.459
Multi-Recording Dev	Whisper Turbo Romani LoRA	trained Whisper adapter	0.769	0.405	0.753	0.393
Multi-Recording Dev	MMS-1B-All Romani/Latin Zero-Shot	external zero-shot comparison	0.756	0.383	0.746	0.376

Prompt Test contains one source group only, so it is useful for continuity with the original model card but too easy to be the only headline result. Multi-Recording Dev spans five recording groups and is a better stress test for iteration.

The detailed frozen report is included in this repository at eval/frozen-asr-evaluation-2026-08-10.md.

Known Limitations

This is an experimental low-resource ASR adapter. It still makes frequent word-boundary, spelling, and diacritic mistakes. It is kept as a reproducible Whisper adapter baseline while MMS adapter training is investigated separately.

The MMS rows above are comparison results only. This repository does not contain an MMS adapter.

Intended Use

Use this adapter for experimentation with Latin-script Romani transcription. Review transcripts before using them in user-facing or high-stakes settings.

Not Included

This repository does not include the base Whisper model, raw audio data, training optimizer state, or training/evaluation logs.