Hindi-English Translator
Model Description
- What it does: Translates Hindi sentences into English.
- Key features: Custom Transformer architecture built from PyTorch primitives. SentencePiece tokenizer. 3-way tie embeddings. RoPE. KV-cache decoding. Beam search with length and coverage penalty.
- Trained by: Rishi Jain.
Architecture Details
- Model Type: Custom Transformer.
- Parameters: ~52M.
- Hidden Size: 512.
- Layers: 6.
- Vocabulary Size: 16,000 (SentencePiece Unigram).
Usage
1# `pip install "transformers<5.0.0"`
2# translation is no longer supported
3# on transformers 5.0.0+
4from transformers import pipeline
5pipe = pipeline("translation", model="Rishi-Jain-27/hindi-english-translator", trust_remote_code=True)
6
7# Load the model.
8from transformers import AutoModel
9model = AutoModel.from_pretrained("Rishi-Jain-27/hindi-english-translator", trust_remote_code=True, dtype="auto")
10
Training Data
- Dataset: IITB English-Hindi parallel corpus.
- Preprocessing: Unicode NFC on both sides. Indic-NLP normalization on Hindi. Train-only filtering: Devanagari/Latin script check per side, source-target length-rato filter, exact-duplicate dedupe, removal of train pair leakage. LaBSE cosine-similarity (threshold to keep ~67% mean cosine 0.73), dropping misaligned pairs. Joint SentencePiece unigram model trained on both sides of filtered train, with byte_fallback. Dev and test normalized but not filtered.
Training Procedure
- Hardware: NVIDIA Tesla T4.
- Hyperparameters: Transformer (d_model=512, n_heads=8, 6 enc/dec layers, pre_norm LayerNorm, ReLU FFN, RoPE, 3-way tied embeddings). AdamW (β₁=0.9, β₂=0.98, ε=1e-9), weight decay on ≥2-D params. Schedule (inverse-sqrt, 4,000-step linear warmup). Label smoothing 01, dropout 01, gradient clipping 1.0, EMA (decay 0.999) evaluated/served, token-based batching by length-bucketed sampling and gradient accumulation, exact token-level loss normalization across micro-batches. Mixed precision (bf16 on A100, fp16 + GradScaler on T4), autodetected.
- Loss Curve/Metrics: Dev NLL: 9.68 -> 1.94 (token perplexity = 6.9).
On test set (2,507 pairs, EMA weights):
| Decoding | BLEU | chrF++ | TER |
|---|
| Greedy (KV-cache) | 19.25 | 48.70 | 69.75 |
| Beam-5 (length penalty 0.6) | 20.07 | 49.08 | 68.28 |
Limitations and Biases
- Known Issues
Trained from scratch. Won't match IndicTran2 / NLLB / mBART on BLEU.
Single-domain corpus: IITB skews to government, news, and religious/literary text.
Translation quality degrades on converstional, technical, or code-switched input.
Sentence-level only: no document context; pronouns, discourse markers, and cross-sentence coreference can be mistranslated.
Length bias: greedy/beam decoding favors shorter outputs; very long source sentences (>~100 subwords) lose content. Length-penalty 0.6 partially mitigates this.
LaBSE filtering bias: ~33% of cleaned train pairs were dropped for low cross-lingual similarity, which can over-represent literal/easy alignments and under-represent free translations.
Inherits corpus biases: gender defaults, named-entity coverage, and dialect/register reflect what IITB contains — primarily Standard Hindi (Devanagari), regional variants.
Hindi→English direction only in this checkpoint. The reverse (en→hi) model exists but was trained for back-translation use, not for end-user translation (dev NLL 2.167, weaker direction).
- Intended Use
Research and educational demonstration of a from-scratch Transformer NMT system. Every sublayer hand-built (no
nn.Transformer / nn.MultiheadAttention, no pretrained weights anywhere).
Hindi→English translation of well-formed, sentence-level Devanagari text in domains close to the IITB distribution.
Component in a speech-to-speech demo (pretrained ASR -> this translator -> pretrained TTS) deployed on HuggingFace Spaces.
Not intended for: production translation, medical/legal/safety-critical text, conversational Hinglish, or any setting where translation errors carry real-world cost.
Acknowledgements
