Views
No views yet
README.md for Hugging Face Model Card1---
2
3datasets:
4 - custom
5library_name: fairseq
6model-index:
7 - name: Malayalam to Hindi Translation (Fairseq)
8 results:
9 - task:
10 name: Translation
11 type: translation
12 dataset:
13 name: Custom Malayalam-Hindi Parallel Corpus
14 type: translation
15 metrics:
16 - name: BLEU
17 type: bleu
18 value: 29.56
19 - name: COMET
20 type: comet
21 value: 0.62
22
23 - name: Hindi to Malayalam Translation (Fairseq)
24 results:
25 - task:
26 name: Translation
27 type: translation
28 dataset:
29 name: Custom Malayalam-Hindi Parallel Corpus
30 type: translation
31 metrics:
32 - name: BLEU
33 type: bleu
34 value: 11.08
35 - name: COMET
36 type: comet
37 value: 0.76
38---
39
40Malayalam ↔ Hindi Translation Model (Fairseq)
41
42This is a **Neural Machine Translation (NMT)** model trained to translate between **Malayalam (ml)** and **Hindi (hi)** using the **Fairseq** framework. It was trained on a custom curated low-resource parallel corpus.
43
44Model Architecture
45
46- Framework: Fairseq (PyTorch)
47- Architecture: Transformer
48- Type: Sequence-to-sequence
49- Layers: 6 encoder / 6 decoder
50- Embedding size: 512
51- FFN size: 2048
52- Attention heads: 8
53- Positional encoding: sinusoidal
54- Tokenizer: SentencePiece (trained jointly on ml-hi)
55- Vocabulary size: 32,000 (joint BPE)
56
57Training Details
58
59| Setting | Value |
60|----------------------|------------------------|
61| Framework | Fairseq (0.12.2) |
62| Training steps | 100k |
63| Optimizer | Adam + inverse sqrt LR |
64| Batch size | 4096 tokens |
65| Max tokens | 4096 |
66| Dropout | 0.3 |
67| BLEU (test set) | 28.5 |
68| Hardware | 1 x V100 32GB GPU |
69| Training time | ~16 hours |
70
71Evaluation
72
73The model was evaluated on a manually annotated Malayalam-Hindi test set consisting of 10,000 sentence pairs.
74
75| Metric | Score |
76| |---------|---------|
77| | hi-ml |ml-hi |
78|-------------------------------|
79| BLEU | 11.08 | 29.56 |
80| COMET | 0.76 | 0.62 |
81
82Usage
83
84In Fairseq (CLI)
85
86```bash
87fairseq-interactive /data-bin \
88 --path checkpoint_best.pt \
89 --task translation_multi_simple_epoch \
90 --lang-pairs hi-ml,ml-hi \
91 --source-lang <src_lang> \
92 --target-lang <tgt_lang> \
93 --batch-size 1 \
94 --beam 10 \
95 --remove-bpe \
96 --lenpen 1.2 \
97 --encoder-langtok src \
98 --decoder-langtok \
99 --skip-invalid-size-inputs-valid-test
1001import torch
2
3# Load model checkpoint
4checkpoint = torch.load('checkpoint_best.pt')
5model.load_state_dict(checkpoint['model_state_dict'])
6model.eval()Note: To use this model effectively, you need the SentencePiece model (spm.model) and the exact Fairseq dictionary files (dict.ml.txt,dict.hi.txt).