A fine-tuned version of openai/whisper-small for Pakistani Punjabi speech recognition in Shahmukhi script (the Perso-Arabic script used to write Punjabi in Pakistan).

Model description

This model transcribes spoken Pakistani Punjabi into written Shahmukhi text. It was fine-tuned from OpenAI's multilingual Whisper Small checkpoint, which had a baseline Word Error Rate (WER) of roughly 70% on this language before fine-tuning.

Because Shahmukhi shares its character set with Urdu, the model uses the Urdu language token (ur) internally rather than the Whisper pa token, which corresponds to Eastern Punjabi in Gurmukhi script. The language and task are baked into the generation config, so you do not need to pass them at inference time.

Results

Evaluation set	WER
Baseline (whisper-small, no fine-tuning)	~70%
Validation (seen speakers)	7.07%
Test (completely unseen speaker)	11.50%

The test WER is measured on a speaker whose voice was never seen during training, which reflects real-world generalization to new voices.

Intended use

Transcription of Pakistani Punjabi audio into Shahmukhi text
Voice interfaces, subtitles, and speech data pipelines for Pakistani Punjabi
A starting checkpoint for further fine-tuning on additional Punjabi data

How to use

python

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
import torch

REPO_ID = "Ghost3454/translynx-pakistani-punjabi-whisper-small"

processor = WhisperProcessor.from_pretrained(REPO_ID)
model     = WhisperForConditionalGeneration.from_pretrained(REPO_ID)
device    = "cuda" if torch.cuda.is_available() else "cpu"
model     = model.to(device).eval()

# load a 16 kHz audio file
audio, _ = librosa.load("sample.wav", sr=16000)

inputs = processor(
    audio, sampling_rate=16000, return_tensors="pt"
).input_features.to(device)

with torch.no_grad():
    generated_ids = model.generate(inputs)

text = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(text)

The language token (ur), task (transcribe), and a repetition guard are already set in the generation config, so no extra generation arguments are required.

Training data

3,500 recordings total
7 speakers, each recording the same 500 Punjabi sentences
Sentences written in Shahmukhi script
One speaker held out entirely as an unseen test set; the remaining six split 90/10 into train and validation

Training procedure

The model was fully fine-tuned (all parameters trained, no LoRA or quantization) on a single consumer GPU.

Key hyperparameters:

Base model: openai/whisper-small
Learning rate: 1e-5 with 100 warmup steps
Effective batch size: 16 (batch size 4 with gradient accumulation of 4)
Mixed precision: fp16
Max steps: 2000, evaluating every 200 steps
Best checkpoint selected by validation WER
Language token forced to ur for stable generation
no_repeat_ngram_size=3 to prevent decoding loops

Limitations

Trained on a controlled set of 500 sentences across 7 speakers, so vocabulary and acoustic diversity are limited compared to large-scale datasets.
Performance may drop on spontaneous speech, heavy background noise, dialects far from the training speakers, or vocabulary outside the training sentences.
Optimized for Shahmukhi output; it is not intended for Gurmukhi (Eastern Punjabi) transcription.

Team & credits

Developed by the TransLynx team:

Ali Hamza — model fine-tuning, training pipeline, and evaluation
Farwa — project planning, data collection, and recording
Jhangir — project planning, data collection, and recording

Thanks to all seven speakers who contributed recordings to the training dataset, without whom this model would not exist.

Citation

If you use this model, please credit the TransLynx Pakistani Punjabi ASR project.