A PEFT LoRA adapter on vinai/PhoWhisper-small that adds robustness to telephony degradation (8 kHz + µ-law + babble + reverb) while keeping PhoWhisper's strong clean Vietnamese accuracy.

Lesson-driven: clean WER on whisper-small was at the FLEURS-vi floor (~0.197), so the lever to reduce both clean and telephony was a stronger Vietnamese base. Swapping to PhoWhisper (clean 0.135) and re-applying the proven recipe — encoder-only LoRA + encoder-consistency loss (anchor adapted encoder ≈ frozen base on clean) + held-out-val selection — reduced both.

Results (FLEURS vi test, beam-5)

Model	Clean WER	Telephony WER	Trainable	Size
Whisper Exp C base	~0.197	0.7419	—	—
LoRA on Exp C (prev best)	0.1972	0.5393	2.15%	21 MB
PhoWhisper-small base	0.1351	0.6109	—	—
This adapter (PhoWhisper + LoRA)	0.1324	0.4664	2.15%	21 MB

vs previous best: clean −33%, telephony −14% (telephony −37% vs original Exp C base). Telephony test = FLEURS vi + telephony_sim (ESC-50 held-out cats 40–49, SNR 0–15).

Usage

python

from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel

base = WhisperForConditionalGeneration.from_pretrained("vinai/PhoWhisper-small")
model = PeftModel.from_pretrained(base, "banhchungtuongot/phowhisper-vi-telephony-lora-encoder")
processor = WhisperProcessor.from_pretrained("vinai/PhoWhisper-small")
processor.tokenizer.set_prefix_tokens(language="vi", task="transcribe")
# decode with num_beams=5 for the reported numbers

Training code & analysis: https://github.com/NguyenDucAnforwork/asr-h100-noise-robust (training/finetune_whisper_tele_lora_v2.py, reports/telephony_lora_finetune.md).

Framework versions

PEFT 0.13.2