Fine-tuned version of openai/whisper-small for Automatic Speech Recognition (ASR) of the Santali language, using a custom Roman-script transliteration scheme called Sanlish.

This model is part of a thesis research pipeline for low-resource Santali speech processing, targeting the task of Santali Speech → Sanlish Transcription → Bangla Translation.

Model Details

Model Description

Developed by: thunderboltc
Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
Language: Santali (sat) — transcribed in custom Roman transliteration (Sanlish)
Base model: openai/whisper-small
Parameters: 241,734,912 (~242M)
Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
License: MIT

What is Sanlish?

Sanlish is a custom Roman-script transliteration of Santali designed to bridge the gap between Santali phonetics and Latin-alphabet-based ASR systems. It was developed as part of this thesis to serve as an intermediate representation between Santali audio and Bangla translation.

Training Details

Training Data

Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
Collected by: thunderboltc (thesis dataset, not yet publicly released)
Split:

Split	Size	Percentage
Train	401	80%
Validation	50	10%
Test	51	10%

Audio format: WAV, 16kHz mono
Text target: Custom Roman transliteration (Sanlish)

Training Procedure

Preprocessing

Audio resampled to 16,000 Hz
Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
Text tokenized using WhisperTokenizer with language=english, task=transcribe
forced_decoder_ids and suppress_tokens disabled to allow custom vocabulary transcription

Training Hyperparameters

Parameter	Value
Epochs	25
Learning Rate	1e-5
Batch Size (per device)	8
Gradient Accumulation Steps	2 (effective batch = 16)
Warmup Steps	50
Precision	fp16 mixed precision
Evaluation Strategy	Per epoch
Best Model Selection	Lowest validation WER
Hardware	NVIDIA Tesla T4 (Google Colab)
Training Duration	~73 minutes

Best Checkpoint

Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).

Evaluation

Results on Test Set (51 utterances)

Metric	Value
Word Error Rate (WER)	56.55%
Character Error Rate (CER)	13.31%

These results are consistent with expectations for a low-resource ASR system trained on ~400 utterances. The large gap between WER and CER reflects that the model captures phonetic character structure well, even when full word sequences deviate from the reference.

Sample Predictions

Reference	Prediction
onde alom chala a	onde alom chala a ✓
tehindo adi raashkaien yaikaoda	tehindo adi reskaien yaikaoda
gapa shetah ale saote baskiyago meh	gapa setah ale saote baskiyago meh
shetay bhuh da	shetay bhuheda

How to Get Started

python

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")

# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)

# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(inputs.input_features)

# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])

Intended Use

Direct Use

Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.

Downstream Use

This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.

Out-of-Scope Use

Not suitable for general-purpose ASR in other languages
Not recommended for standard Santali Ol Chiki script transcription
Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set

Limitations & Bias

Trained on only 502 utterances — a very small dataset by ASR standards
May not generalize well to unseen speakers or recording environments
Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
No speaker diversity information available for the training set

Citation

If you use this model in your research, please cite:

bibtex

@misc{thunderboltc2025whisper-santali,
  author = {thunderboltc},
  title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}

Environmental Impact

Hardware: NVIDIA Tesla T4
Cloud Provider: Google Colab
Hours used: ~1.25 hours
Carbon emissions: Estimated using the ML Impact Calculator