You can deploy this model and start earning money today!

eluve-inc/
SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7

Name: SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7
Availability: InStock

safetensors

modernbert

multilingual

Views

No views yet

SEA-LION ModernBERT 600M Multilingual NLI 26lang 2mil7

SEA-LION ModernBERT 600M Multilingual NLI 26lang 2mil7 is a natural language inference (NLI) model based on the ModernBERT architecture. The model was originally the pre-trained aisingapore/SEA-LION-ModernBERT-600M model, which is then fine-tuned on MoritzLaurer/multilingual-NLI-26lang-2mil7. The model is then evaluated zero-shot on IndoNLI.

Evaluation Results

Zero-shot accuracy on IndoNLI:

	`dev` Acc.	`test_lay` Acc.	`test_expert` Acc.
`IndoNLI`	84.30	82.05	77.11

How to Use

As Natural Language Inference (NLI) Model

python

from transformers import pipeline

model_name = "eluve-inc/SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7"

clf = pipeline("text-classification", model=model_name, tokenizer=model_name, top_k=None, device="cuda:0")

premise = "Minggu lalu saya mulai batuk pilek, dan sekarang saya sudah sembuh."
hypothesis = "Pasien sudah tidak lagi mengalami gejala batuk pilek."

out = clf({"text": premise, "text_pair": hypothesis})
# [{'label': 'entailment', 'score': 0.8801426887512207},
#  {'label': 'neutral', 'score': 0.08782942593097687},
#  {'label': 'contradiction', 'score': 0.03202790021896362}]

Model

Model	#params	Arch.	Training/Validation data (text)
`SEA-LION-ModernBERT-600M-multilingual-NLI-26lang-2mil7`	600M	ModernBERT	Multilingual NLI (26 languages)

Model Configuration

labels: entailment (0), neutral (1), contradiction (2)

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 256
gradient_accumulation_steps: 1
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
warmup_ratio: 0.1
weight_decay: 0.01
num_epochs: 3
mixed_precision_training: bf16

References

[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). IndoNLI: A Natural Language Inference Dataset for Indonesian. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

[2] Warner, B., Chaffin, A., Clavié, B., Weller, O., Hallström, O., Taghadouini, S., Gallagher, A., Biswas, R., Ladhak, F., Aarsen, T., Cooper, N., Adams, G., Howard, J., & Poli, I. (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. arXiv preprint arXiv:2412.13663.