Hindi-English Code-Switch ASR (data2vec-AQC + Gemma-3-4B-IT LoRA)
A bilingual automatic speech recognition (ASR) model for Hindi, English, and Hindi-English code-switched speech, built on the SLAM-LLM framework. It combines a frozen data2vec-AQC speech encoder, a trainable linear projector, and google/gemma-3-4b-it fine-tuned with LoRA.
This checkpoint (epoch2_step146286) was selected as the overall best checkpoint by validation loss across the full training run, and independently confirmed as the strongest or most balanced checkpoint across 9 held-out Hindi/code-switch benchmark testsets (6 out of 9 wins vs. neighboring checkpoints — see comparison table below).
Only the projector + LoRA adapter weights are released here (~117MB) — the base data2vec-AQC encoder and google/gemma-3-4b-it weights are not included and must be obtained separately (see Usage below).
Training Data
2,683,431 utterances total from a bilingual Hindi-English code-switch corpus
~4000 hours Hindi speech, ~1000 hours English speech, with code-switch (hi-en) tagged utterances
Prompt format follows Gemma's chat template with a rolling previous-utterance-context field for long-form continuity
Checkpoint Comparison (WER %, lower is better)
WER computed on 9 held-out Hindi/code-switch benchmark testsets, using an edit-distance-based scorer with NFC normalization and punctuation stripped from both reference and hypothesis (standard WER convention).
Testset
Epoch 1 (step 151000)
Epoch 2 (step 146286)
Epoch 3 best (step 136572)
Epoch 3 latest (step 167572)
CommonVoice (Hindi)
14.98
12.41
13.13
13.42
FLEURS (Hindi)
14.82
12.70
13.31
14.23
IndicTTS (Hindi)
11.08
11.63
12.27
11.72
Kathbath Noisy
10.80
10.29
10.63
10.98
Kathbath
10.42
9.76
10.03
10.39
MUCS
11.21
10.93
11.10
11.00
Gramvaani
26.09
22.99
21.87
20.49
Vendor R1 (code-switch)
19.25
18.45
18.23
18.87
Vendor R2 (code-switch)
20.21
19.15
18.88
19.48
This checkpoint (Epoch 2, step 146286) wins 6 of 9 testsets and is the most balanced overall.
Comparison vs. Meta's Omnilingual ASR (omniASR_LLM_7B_v2)
This model (a specialized ~30M-trainable-parameter bilingual fine-tune) outperforms Meta's general-purpose 7B multilingual ASR model on 8 of 9 Hindi/code-switch benchmarks:
Testset
This model (Epoch 2)
Omni-ASR LLM 7B v2
CommonVoice (Hindi)
12.41
16.63
FLEURS (Hindi)
12.70
11.24
IndicTTS (Hindi)
11.63
13.55
Kathbath Noisy
10.29
15.00
Kathbath
9.76
13.48
MUCS
10.93
15.60
Gramvaani
22.99
30.44
Vendor R1 (code-switch)
18.45
28.62
Vendor R2 (code-switch)
19.15
30.52
Note: Omnilingual ASR has no dedicated Hindi-English code-switch language code — each utterance is forced into a single target script (hin_Deva or eng_Latn), which likely disadvantages it on the code-switch-heavy testsets (Vendor R1/R2) shown above. This model was trained specifically for code-switch conditioning.
Usage
This is not a plain transformers-loadable checkpoint — it requires the SLAM-LLM inference pipeline plus the two frozen base models:
See config.json in this repo for the full architecture/hyperparameter spec.
Prompt Format
<start_of_turn>user
Transcribe speech to {lang} text.{lang_note} If available, use this previous context: {prev_context}<end_of_turn>
<start_of_turn>model
Where {lang} is Hindi, English, or Hindi-English, and {lang_note} adds " This utterance code-switches between Hindi and English." for code-switch utterances.
License
Apache 2.0. Note that the base google/gemma-3-4b-it model is subject to its own Gemma license — this release only contains the LoRA adapter and projector deltas, not the base model weights.
Citation
If you use this model, please cite the SLAM-LLM framework and this release.
