This is a LoRA adapter for
openai/whisper-small fine-tuned on Jiangyin dialect from
WuSutra.com.
Wusutra.com is a dialect crowdsourcing website which implements the
entire ML workflow — including audio upload, model training, validation, and inference.
You can upload your own audios and even trigger the training yourself on wusutra.com. If you have further questions, feel free to message me.
1import torch
2from transformers import WhisperForConditionalGeneration, WhisperProcessor
3from peft import PeftModel
4
5# Load base model
6model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
7processor = WhisperProcessor.from_pretrained("openai/whisper-small")
8
9# Load LoRA adapter
10model = PeftModel.from_pretrained(model, "jxue/whisper_small_jiangyin_lora")
11
12# Transcribe audio
13with torch.no_grad():
14 input_features = processor(
15 audio_array, sampling_rate=16000, return_tensors="pt"
16 ).input_features.to(device)
17
18 predicted_ids = model.generate(
19 input_features=input_features,
20 forced_decoder_ids=processor.get_decoder_prompt_ids(language="zh", task="transcribe")
21 )
22
23transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
