You can deploy this model and start earning money today!

manish320/
whisper-small-jv-id-lora

Name: whisper-small-jv-id-lora
Rating: 2.5 (48 reviews)

safetensors

whisper

Views

No views yet

Whisper Small — Indonesian & Javanese Code-Switching ASR

This model is a fine-tuned version of OpenAI Whisper Small for automatic speech recognition (ASR) on Indonesian and Javanese speech, including code-switched utterances.

Model Details

Base model: openai/whisper-small
Task: Automatic Speech Recognition (ASR)
Languages: Indonesian (id) and Javanese (jv)
Training method: LoRA (Low-Rank Adaptation)
Final model: LoRA adapter merged into the base Whisper model
Model type: Encoder-decoder Transformer
Framework: Hugging Face Transformers
Audio sampling rate: 16 kHz

Intended Use

This model is intended for speech-to-text transcription of Indonesian and Javanese audio.

It is particularly useful for:

Indonesian speech recognition
Javanese speech recognition
Indonesian–Javanese code-switching
Conversational speech transcription
Multilingual ASR experiments

Training Approach

The model was fine-tuned using LoRA (Low-Rank Adaptation) instead of updating all parameters of the Whisper model.

The training process was:

text

OpenAI Whisper Small
        ↓
LoRA Fine-Tuning
        ↓
Indonesian + Javanese Speech Data
        ↓
Code-Switched ASR Training
        ↓
LoRA Adapter
        ↓
Adapter Merging
        ↓
Final Merged Whisper Model

The final uploaded model contains the merged weights, so the LoRA adapter does not need to be loaded separately.

Why LoRA?

LoRA allows efficient fine-tuning by training only a small number of additional parameters while keeping most of the original Whisper parameters frozen.

Advantages include:

Lower GPU memory requirements
Faster fine-tuning
Fewer trainable parameters
Efficient experimentation
Easy adapter-based training

After training, the LoRA weights were merged with the original Whisper model to create a standalone model.

Model Architecture

The model is based on the Whisper encoder-decoder architecture:

text

Audio
  ↓
Whisper Feature Extractor
  ↓
Whisper Encoder
  ↓
Whisper Decoder
  ↓
Text Transcription

Usage

Installation

pip install -U transformers torch librosa soundfile

Load the Model

python

import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration

model_id = "manish320/whisper-small-jv-id-lora"

processor = WhisperProcessor.from_pretrained(model_id)

model = WhisperForConditionalGeneration.from_pretrained(model_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

Transcribe Audio

python

import librosa
import torch

audio, sampling_rate = librosa.load(
    "audio.wav",
    sr=16000
)

inputs = processor(
    audio,
    sampling_rate=16000,
    return_tensors="pt"
)

input_features = inputs.input_features.to(device)

with torch.no_grad():
    predicted_ids = model.generate(input_features)

transcription = processor.batch_decode(
    predicted_ids,
    skip_special_tokens=True
)[0]

print(transcription)

Example

Input audio:

Saya sedang pergi ke pasar, banjur tuku panganan.

Possible transcription:

Saya sedang pergi ke pasar, banjur tuku panganan.

The model is designed to preserve the language switching present in the speech rather than forcing the complete utterance into a single language.

Language / Decoding

For multilingual and code-switching speech, language forcing can be avoided so that Whisper can generate the transcription naturally.

For example:

python

predicted_ids = model.generate(
    input_features
)

If you specifically want to force a language, Whisper generation options can also be configured depending on the evaluation setup.

Limitations

The model may have reduced performance on:

Languages other than Indonesian and Javanese
Heavy background noise
Very low-quality recordings
Strong accents not represented in the training data
Speakers or domains significantly different from the training dataset
Very long audio recordings
Code-switching patterns that are underrepresented in the training data

Performance may also vary depending on microphone quality, speaking style, audio duration, and domain.

Evaluation

Recommended ASR evaluation metrics include:

Word Error Rate (WER)
Character Error Rate (CER)

Example:

python

from jiwer import wer

reference = "reference transcription"
prediction = "model transcription"

score = wer(reference, prediction)

print("WER:", score)

Lower WER indicates better transcription performance.

Base Model

This model is based on:

OpenAI Whisper Small

Whisper is a multilingual automatic speech recognition model trained on a large-scale collection of audio and text data.

Model Files

The repository contains the merged model weights and configuration required to load the model with Hugging Face Transformers.

Typical files include:

text

config.json
generation_config.json
model.safetensors
preprocessor_config.json
tokenizer_config.json
special_tokens_map.json
vocab.json
merges.txt

License

Please refer to the license of the original Whisper model and the license/terms of the dataset used for fine-tuning.

Acknowledgements

Thanks to:

OpenAI for the Whisper model
Hugging Face for the Transformers and PEFT ecosystem
The creators and contributors of the Indonesian and Javanese speech datasets used for fine-tuning

Citation

If you use this model in your project, please cite the original Whisper work and this model repository.

text

Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

Radford et al., 2022

Disclaimer

This model is provided for research and development purposes. Transcription quality may vary depending on the audio, speaker, language, accent, and domain.

license: mit language:

jv, id, multilingual metrics:
wer




---
license: mit
base_model:
- openai/whisper-small
pipeline_tag: automatic-speech-recognition
tags:
- indonesian
---