MyNLI: Burmese Natural Language Inference with XLM-RoBERTa
A Burmese Natural Language Inference (NLI) model fine-tuned from xlm-roberta-base
This model predicts the relationship between a premise and a hypothesis as one of:
Entailment Neutral Contradiction
Model Details
Base model: xlm-roberta-baseLanguage: Burmese (Myanmar)Task: Natural Language Inference (NLI)Labels: entailment, neutral, contradictionFramework: Transformers / PyTorch
Dataset
The model is trained on an ~8K Burmese NLI dataset , prepared from:
Cleaned Burmese NLI data (source: [(https://huggingface.co/datasets/akhtet/myanmar-xnli )] )
Additional manually created samples
Translated English NLI data for diversity
Dataset Structure
Most samples follow a 1 premise → 3 hypotheses structure
Each hypothesis has a different NLI label
An additional genre
Intended for future zero-shot / cross-genre experiments
Not used during training yet
Preprocessing
Since a pretrained multilingual LLM is used, no manual tokenization (word-level or syllable-level) is applied.
Steps:
Unicode normalization (NFC)Zawgyi detection Automatic conversion to Unicode if Zawgyi text is detectedRely on XLM-R subword tokenizer for tokenization
Data Splitting Strategy
To prevent data leakage caused by shared premises:
Train: 70%Validation: 15%Test: 15%
Instead of random shuffling:
GroupShuffleSplit is used
Samples with the same premise always stay in the same split
Prevents:
Premise overlap across splits
Hypothesis leakage between train / validation / test sets
Training Setup
Epochs: 4Learning rate: 2e-5Batch size: 8Weight decay: 0.01Warmup ratio: 0.1FP16 training Best model selected by: Validation F1 score Seed: 42
Evaluation Metrics
The model is evaluated using:
Training Results
Epoch Train Loss Val Loss Accuracy F1 1 0.9509 0.8948 0.5602 0.5143 2 0.7850 0.6888 0.7233 0.7153 3 0.6067 0.6367 0.7660 0.7603 4 0.4301 0.7060 0.7455 0.7411
Best checkpoint selected based on F1 score .
Test Set Performance
1 {
2 "eval_loss" : 0.7713 ,
3 "eval_accuracy" : 0.7868 ,
4 "eval_f1" : 0.7852
5 }
Confusion Matrix on Test Set
[[352 57 38]
[ 53 270 34]
[ 27 46 319]]Rows represent true labels , columns represent predicted labels
(Label order: entailment, neutral, contradiction)
Inference Example
You can use the model as follows:
1 from transformers import AutoTokenizer , AutoModelForSequenceClassification
2 import torch
3
4 model_name = "emilyyy04/xlm-roberta-base-burmese-nli"
5
6 tokenizer = AutoTokenizer . from_pretrained ( model_name )
7 model = AutoModelForSequenceClassification . from_pretrained ( model_name )
8
9 premise = "သူမသည် ဆေးရုံတွင် ဆရာဝန်အဖြစ် အလုပ်လုပ်နေသည်။"
10 hypothesis = "သူမသည် ကျန်းမာရေးလုပ်ငန်းတွင် အလုပ်လုပ်နေသည်။"
11
12 inputs = tokenizer (
13 premise ,
14 hypothesis ,
15 return_tensors = "pt" ,
16 truncation = True ,
17 padding = True
18 )
19
20 outputs = model ( ** inputs )
21 predicted_class = torch . argmax ( outputs . logits , dim = 1 ) . item ( )
22
23 label_map = { 0 : "entailment" , 1 : "neutral" , 2 : "contradiction" }
24 print ( "Predicted label:" , label_map [ predicted_class ] )
25 # conf
26 probs = torch . softmax ( outputs . logits , dim = - 1 ) [ 0 ]
27 print ( "Confidence:" , { k : round ( float ( probs [ i ] ) , 3 ) for i , k in label_map . items ( ) } )
28
29
Limitations & Future Work
Genre-aware and zero-shot classification is planned but not yet implemented
Performance may vary for:
Very long inputs
Out-of-domain or highly informal Burmese
Future improvements:
Larger native Burmese NLI dataset
Explicit genre-based evaluation
Domain adaptation
