Pnar-English Machine Translation
This model is a fine-tuned version of
facebook/nllb-200-distilled-600M
for
Pnar (ISO 639-3: pbv) ↔ English machine translation.
The model corresponds to the Gold + Silver (Model C) condition described in:
When Silver Isn't Enough: Data Quality Effects on Pnar-English Machine Translation
The model was trained using a combination of a manually translated Gold corpus and automatically generated Silver corpus. The combined dataset produced the strongest overall performance among the evaluated NLLB-200 configurations.
Model Details
- Base model: NLLB-200 distilled 600M
- Parameters: ~600M
- Languages: Pnar ↔ English
- Pnar language code:
pnar_Latn
- English language code:
eng_Latn
- Architecture: NLLB-200
- Training: Fine-tuning
- Optimizer: AdamW
- Learning rate:
5e-5
- Batch size: 8
- Maximum sequence length: 128 tokens
- Precision: FP16
- Epochs: 5
- GPU: NVIDIA A40
The Pnar language token was added to the tokenizer and model embedding matrix for fine-tuning.
Training Data
Gold Data
The Gold corpus contains 4,826 Pnar-English training pairs.
English source sentences were obtained from the Tatoeba corpus and translated into Pnar by two paid native Pnar translators. All translations were human-verified.
Silver Data
The Silver corpus contains 48,519 training pairs derived from the Pnar-English portion of FineTranslations.
The source Pnar text originates primarily from Wyrta, a regional Pnar news media outlet. The English translations were synthetically generated using Gemma 3 27B.
The Silver data therefore primarily represents the news domain.
Combined Dataset
| Split | Gold | Silver | Combined |
|---|
| Train | 4,826 | 48,519 | 53,345 |
| Validation | 604 | 6,012 | 604 |
| Test | 604 | 5,937 | 604 |
The Gold test set is used as the common evaluation benchmark.
Evaluation
Automatic Evaluation
Evaluation was performed on the Gold test set.
| Direction | BLEU | ChrF | TER | COMET |
|---|
| English → Pnar | 30.39 | 53.29 | 52.74 | 0.693 |
| Pnar → English | 26.02 | 46.75 | 60.37 | 0.700 |
BLEU, ChrF, TER and COMET were computed using the Hugging Face evaluate library.
COMET should be interpreted as a complementary neural metric because its underlying reference models have not been validated specifically for Pnar.
Human Evaluation
Human evaluation was conducted on 50 English → Pnar translations from the Gold test set.
Two native Pnar speakers independently evaluated outputs on a 1–5 scale.
| Metric | Score |
|---|
| Adequacy | 4.25 / 5 |
| Fluency | 4.53 / 5 |
| Quadratic weighted κ — Adequacy | 0.70 |
| Quadratic weighted κ — Fluency | 0.91 |
The results indicate generally adequate meaning transfer and fluent Pnar output.
Intended Use
This model is intended for:
- English ↔ Pnar machine translation
- Research on low-resource machine translation
- Pnar language technology development
- Evaluation of multilingual MT systems for Northeast Indian languages
- Development of Pnar-language applications and datasets
Limitations
The model should not be treated as a fully reliable translation system.
The Gold training corpus is relatively small, with only 4,826 training pairs. The Gold and Silver datasets also originate from different domains, meaning that the observed advantage of Gold data may reflect both data quality and domain similarity.
The model may perform poorly on:
- Idiomatic expressions
- Domain-specific terminology
- Long or complex sentences
- Text substantially outside the training domains
- Ambiguous expressions
Human evaluation found examples where the output was fluent but did not fully preserve the meaning of the English source.
The experiments used a fixed hyperparameter configuration and a single training run per condition. Multi-seed experiments and different Gold/Silver mixing ratios were not evaluated.
Bias, Risks and Safety
Pnar is a low-resource language with limited publicly available NLP resources. Model outputs may contain mistranslations, omissions, hallucinations, or inappropriate lexical choices.
For high-stakes applications, translations should be reviewed by a qualified Pnar speaker.
Citation
If you use this model, please cite:
1@article{tekcham2026silver,
2 title={When Silver Isn't Enough: Data Quality Effects on Pnar-English Machine Translation},
3 author={Tekcham, Riya and Asma, Fitha and Sulfeekhar, Badal Nyalang},
4 year={2026}
5}
Acknowledgement
This work was supported by MWire Labs, which provided computational resources and support for the research, including assistance with data creation and annotation.
