You can deploy this model and start earning money today!

eulogik/
polywhisper

Name: polywhisper
Availability: InStock

automatic-speech-recognition

Views

No views yet

GitHub

PolyWhisper — Efficient Multilingual Indic ASR

PolyWhisper is an efficient multilingual Automatic Speech Recognition (ASR) system for Indian languages. It uses a frozen Whisper backbone with per-language LoRA adapters to achieve competitive accuracy at a fraction of the compute cost of full fine-tuning.

What is PolyWhisper?

PolyWhisper is a lightweight, production-ready ASR model for 5 major Indian languages: Hindi, Tamil, Telugu, Bengali, and Marathi. Unlike traditional approaches that fine-tune the entire model, PolyWhisper keeps the pretrained Whisper backbone frozen and trains small LoRA adapters (rank 16, ~14MB per language) that bolt onto the backbone at inference time.

Try the Demo

🌐 Live Demo on HuggingFace Spaces — upload audio or record from your microphone

Key Features

5 Indian languages in a single model: Hindi (hi), Tamil (ta), Telugu (te), Bengali (bn), Marathi (mr)
Tiny adapter footprint: ~14MB per language vs ~1.5GB for full fine-tune
Production-quality Hindi & Tamil: WER 37.2 (Hindi) and 60.4 (Tamil) on FLEURS benchmark
Multiple backends: PyTorch (MPS/CUDA/CPU) + ONNX Runtime (CPU-only)
CLI + Python API: polywhisper transcribe audio.wav --lang hi or from polywhisper import transcribe
SRT subtitles: Export transcriptions as timestamped subtitles
Batch processing: Transcribe entire folders of audio files
ONNX export: Quantized INT8 models for CPU deployment

Quick Start

Install

pip install -e .

Requirements: Python 3.9+, PyTorch, Transformers, SoundFile. For CPU-only: pip install onnxruntime.

Transcribe an Audio File

bash

# Hindi
polywhisper transcribe audio.wav --lang hi

# Tamil with JSON output
polywhisper transcribe audio.wav --lang ta --format json

# Auto-detect language
polywhisper transcribe audio.wav

Python API

python

from polywhisper import transcribe

# Single file
result = transcribe("audio.wav", lang="hi")
print(result.text)          # "नमस्ते, आप कैसे हैं?"
print(result.segments)      # [Segment(text=..., start_sec=0.0, end_sec=5.2)]

# Long audio (auto-chunks into 30s segments)
result = transcribe("long_meeting.mp3", lang="te", max_new_tokens=512)

Batch Transcription

polywhisper batch ./audio_folder/ --lang ta --output results.json

SRT Subtitles

polywhisper transcribe video_audio.wav --lang hi --format srt > subtitles.srt

Supported Languages

Language	Code	Script	FLEURS WER	Status
Hindi	`hi`	Devanagari	37.2	Production-ready
Tamil	`ta`	Tamil	60.4	Production-ready
Telugu	`te`	Telugu	105.7	Retraining scheduled
Bengali	`bn`	Bengali	196.3	Retraining scheduled
Marathi	`mr`	Devanagari	167.3	Retraining scheduled

Hindi and Tamil achieve better results than full model fine-tuning (expert baseline) using only ~14MB adapters. Telugu, Bengali, and Marathi are being retrained with domain-matched data for significant improvements.

Performance Comparison

Model	Params	Hindi WER	Tamil WER	Telugu WER
Whisper-Base (vanilla)	74M	131.3	93.2	185.6
Whisper-Small (vanilla)	244M	62.3	68.8	129.6
Whisper-Medium (vanilla)	769M	35.4	49.1	110.3
PolyWhisper Expert (Base+LoRA)	74M+3.5M	38.6	74.2	91.9
PolyWhisper Product (Small+LoRA)	244M+3.5M	37.2	60.4	105.7

Lower WER is better. Evaluated on FLEURS test set with punctuation-normalized scoring.

Model Architecture

┌─────────────────────────────────────────┐
│           Whisper Backbone (frozen)      │
│     244M params (encoder + decoder)     │
├─────────────────────────────────────────┤
│  LoRA Adapter (per-language, 14MB each) │
│  Rank 16, applied to:                   │
│  • Self-attention (Q, K, V, output)     │
│  • Cross-attention (Q, K, V, output)    │
│  • Encoder attention (HI only)          │
└─────────────────────────────────────────┘

Backbone: OpenAI Whisper Small (244M parameters, frozen)
Adapters: Low-Rank Adaptation (LoRA) with rank 16
Total parameters: 244M (backbone) + 3.5M (adapter per language)
Adapter size: ~14MB per language (vs ~1.5GB for full fine-tune)

Export to ONNX (CPU Deployment)

bash

# Export all languages
polywhisper export --lang hi --variant prod --int8
polywhisper export --lang ta --variant prod --int8
polywhisper export --lang te --variant prod --int8
polywhisper export --lang bn --variant prod --int8
polywhisper export --lang mr --variant prod --int8

ONNX models are ~14MB per language with INT8 quantization. Use onnxruntime for CPU-only inference without PyTorch.

Training Details

Training data: IndicVoices-ST (conversational speech)
Evaluation: FLEURS (read speech)
Optimizer: AdamW with cosine LR schedule (1e-4 peak)
Training: 3 epochs per language, batch size 4, 2× NVIDIA T4 GPUs
Techniques: Scheduled sampling, LoRA rank 16, encoder+decoder adaptation

Citation

bibtex

@misc{polywhisper2026,
  title={PolyWhisper: Efficient Multilingual Indic ASR via Frozen Backbone + Per-Language LoRA},
  author={Eulogik Developer},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/eulogik/polywhisper}
}

License

Apache 2.0. See LICENSE for details.

eulogik/polywhisper

PolyWhisper — Efficient Multilingual Indic ASR

What is PolyWhisper?

Try the Demo

Key Features

Quick Start

Install

Transcribe an Audio File

Python API

Batch Transcription

SRT Subtitles

Supported Languages

Performance Comparison

Model Architecture

Export to ONNX (CPU Deployment)

Training Details

Citation

License

Links

eulogik/
polywhisper