LegalLoRA-IndicTrans2-en_indic
This model is a LoRA fine-tuned version of ai4bharat/indictrans2-en-indic-1B on 80,000 high-quality English-Hindi legal sentence pairs. The training data consists of parallel sentences from Indian Supreme Court and Punjab & Haryana High Court orders.
Model Details
- Base model:
ai4bharat/indictrans2-en-indic-1B
- LoRA fine-tuning applied to:
q_proj, k_proj
- Task: English to Hindi machine translation (legal domain)
- Training steps: 40,000
- Best checkpoint: Step 20,000
- Eval set: 8,000 samples (10% of training data)
Evaluation (During Training)
Evaluated on a development set (Dev Set) derived from the training corpus (prior to training); 8,000 pairs (10%)
Metrics at best checkpoint (20000 steps):
- BLEU: 38.26
- chrF++: 54.59
- Eval Loss: 1.98
Sample predictions during Eval
Prediction: न्यायालय ने आगे कहा कि प्रक्रिया भी प्राकृतिक न्याय के सिद्धांतों के अनुरूप होनी चाहिए ।
Reference: न्यायालय ने आगे कहा कि प्रक्रिया प्राकृतिक न्याय के सिद्धांतों के अनुरूप भी होनी चाहिए ।
Real-World Document-Level Evaluation (Post-Training)
To assess real-world performance, we evaluated the model on 100 randomly selected legal documents that were excluded from the training and development datasets. Each sample contained:
original_content_{id}.txt: Original English legal documenttranslated_content_{id}.txt: Human-translated Hindi versionmodel_output_{id}.txt: Hindi translation generated by the modelresults_{id}.json: File containing similarity scores per document
Evaluation Method
We used LaBSE (Language-agnostic BERT Sentence Embedding) to compute semantic similarity between:
original English document ↔ human Hindi translationoriginal English document ↔ model's Hindi output
The final metric represents how close the model's translation is to the meaning conveyed in the original English document.
Result
- Average Semantic Similarity Score (Model Output vs. Original English in the document):
0.70
This indicates that the model retains approximately 70% of the semantic meaning when translating legal English documents into Hindi in a zero-shot, full-document setting.
Evaluation Artifacts
All raw files and result data can be found here:
Evaluation Data & ResultsThe folder includes:
- All document-level files (
original_content, translated_content, model_output)
- Individual evaluation results per document (
results_{id}.json)
- Full summary across all 100 documents in
all_evaluation_results.json
Dataset Curation
The dataset was curated using publicly available bilingual judgments from the Supreme Court of India and the Punjab & Haryana High Court. Original English and Hindi PDF orders were parsed using OCR. Sentence alignment was performed using SentAlign, and pairs with alignment similarity score greater than 0.6 were selected. The final dataset consists of 80,000 English-Hindi aligned legal sentence pairs.
Training Configuration
Training was performed using the IndicTrans2 hugging_face LoRA fine-tuning pipeline with the following parameters:
1python3 train_lora.py \
2 --data_dir en-indic-exp/ \
3 --model ai4bharat/indictrans2-en-indic-1B \
4 --output_dir ./output-lora-finetuned\
5 --src_lang_list eng_Latn \
6 --tgt_lang_list hin_Deva \
7 --save_steps 20000 \
8 --max_steps 40000 \
9 --batch_size 1 \
10 --grad_accum_steps 32 \
11 --warmup_steps 4000 \
12 --max_grad_norm 1.0 \
13 --learning_rate 2e-4 \
14 --adam_beta1 0.9 \
15 --adam_beta2 0.98 \
16 --optimizer adamw_torch \
17 --lr_scheduler inverse_sqrt \
18 --num_workers 0 \
19 --weight_decay 0.01 \
20 --metric_for_best_model eval_BLEU \
21 --greater_is_better \
22 --patience 10 \
23 --lora_target_modules "q_proj,k_proj" \
24 --lora_dropout 0.1 \
25 --lora_r 16 \
26 --lora_alpha 32 \
27 --print_samples
Inference
To run inference using this LoRA fine-tuned checkpoint, run the following github repo.
Intended Use
This model is intended for English to Hindi legal translation tasks. It performs best on judicial or government domain text. It is not recommended for casual or general-purpose translation applications.
Limitations
- The model is fine-tuned solely on legal texts, and may also generalize well outside this domain.
- Evaluation is only performed on in-domain data and a 100-document sample.
- Translation quality may vary depending on the complexity of legal expressions and OCR quality of source documents.
Citation
1@misc{ LegalLoRA-IndicTrans2-en_indic,
2 title={LoRA Fine-tuned IndicTrans2 1B for English to Hindi Legal Translation},
3 author={Aman Dogra},
4 year={2025},
5 url={https://huggingface.co/axondendriteplus/LegalLoRA-IndicTrans2-en_indic}
6}
Acknowledgements
