Fine-tuned version of nvidia/nemotron-3.5-asr-streaming-multilingual-0.6b on the Noisy Hinglish ASR dataset for improved Hindi and code-switched Hinglish speech recognition.

Model Details

Property	Value
Base Model	`nvidia/nemotron-3.5-asr-streaming-multilingual-0.6b`
Architecture	FastConformer RNNT with Prompt-Streaming
Parameters	638M
Encoder	24-layer Conformer, d_model=1024, 8-head attention
Decoder	2-layer LSTM, pred_hidden=640
Joint	RNNT Joint, joint_hidden=640
Vocabulary	13,087 BPE tokens + blank (13088)
Language Prompts	128 (covering 100+ languages)
Context Size	Left: 56, Right: 3 (320ms balanced streaming)
Subsampling	Factor 8 (dw_striding)
Preprocessor	128 Mel filters, 16kHz, 25ms window, 10ms stride
Training	Full fine-tune (all params trainable), encoder frozen
Training Steps	12,000 steps (~8,124 best step)
Hardware	RTX 5070 Ti (16 GB)
License	Apache 2.0

Performance

Benchmark Results

Split	Samples	WER%	CER%	FTR%	Δ WER vs Base
Clean Hindi	500	24.75	10.54	0.80	+2.85 (regression)
Conversational Hinglish	1,036	24.57	17.69	0.97	-17.60
Noisy Hindi	250	31.67	32.33	11.60	-7.46
Negatives (noise)	200	0.00	0.00	99.50	+99.50

Key Improvements

Hinglish WER reduced from 42.17% → 24.57% (-17.6 pp), matching Qwen-ft V2 performance
Noisy Hindi WER reduced from 39.13% → 31.67% (-7.46 pp)
FTR (False Trigger Rate) on background noise: 99.5% rejection (from 0% baseline)
Slight regression on clean Hindi (+2.85 pp) due to model learning Hinglish code-switching patterns

Repository Structure

.
├── README.md                  # This file
├── config.json                # Deployment configuration
├── tokenizer.model            # BPE tokenizer
├── model/
│   ├── nemotron-3.5-hinglish.nemo   # NeMo checkpoint (full model)
│   ├── encoder.onnx                  # ONNX encoder for deployment
│   ├── decoder_joint.onnx            # ONNX decoder+joint for deployment
│   └── *.weight / onnx__*            # External ONNX weight files
├── scripts/
│   ├── export_onnx.py                # ONNX export from .nemo
│   ├── evaluate_finetuned.py         # Benchmark evaluation
│   ├── finetune_frozen_encoder.py    # Training script
│   ├── run_finetune.sh               # Training launcher
│   ├── prepare_training_data.py      # Data preparation
│   └── unpack_dataset.py             # Dataset extraction
└── results/
    ├── finetuned_benchmark_results.md
    └── nemotron_baseline_results.md

Usage

NeMo Inference (Python)

python

import torch
from nemo.collections.asr.models import EncDecRNNTBPEModelWithPrompt

model = EncDecRNNTBPEModelWithPrompt.restore_from("model/nemotron-3.5-hinglish.nemo")
model.eval()

# Transcribe with language prompt
transcription = model.transcribe(
    ["path/to/audio.wav"],
    batch_size=1,
    target_lang="hi-IN",
    prompt_mode="langID",
)[0]
print(transcription)

ONNX Deployment

The model/ directory contains exported ONNX models suitable for production deployment:

encoder.onnx: Streaming encoder with cache-aware inference
decoder_joint.onnx: RNN-T decoder + joint network

bash

# Export from .nemo yourself (optional)
python scripts/export_onnx.py

Streaming Configuration (config.json)

json

{
  "streaming": {
    "chunkMs": 320,
    "chunkSize": 4,
    "rightContext": 3,
    "lookaheadMs": 240,
    "melFrames": 32,
    "preCacheSize": 9,
    "outputFrames": 4
  }
}

Training Details

The model was fine-tuned using NVIDIA NeMo toolkit with the following setup:

Base model: nvidia/nemotron-3.5-asr-streaming-multilingual-0.6b
Dataset: ~23,500 training samples, ~2,300 validation samples from Noisy Hinglish ASR
Optimizer: NovoGrad with learning rate 0.1, cosine schedule with warmup
Precision: BF16 mixed precision
Encoder frozen: Yes (decoder + joint trained)
Loss: RNNT with fastemit lambda 0.005

Citation

bibtex

@misc{nemotron-3.5-hinglish-2026,
  author = {Aditya},
  title = {Nemotron-3.5 Hinglish Fine-Tuned ASR},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/addyo07/nemotron-3.5-0.6b-hinglish}
}