AraModernBert For Topic Classification

Overview

[!NOTE] This is an Experimental Arabic Model demonstrates how ModernBERT can be adapted to Arabic for tasks like topic classification.

• Dataset: Arabic Wikipedia
• Size: 1.8 GB
• Tokens: 228,788,529 tokens

Model Eval Details

• Epochs: 3
• Evaluation Metrics:
  • F1 Score: 0.95
  • Loss: 0.1998
• Training Step: 47,862

Dataset Used For Training:

• SANAD DATASET was used for training and testing which contains 7 different topics such as Politics, Finance, Medical, Culture, Sport , Tech and Religion.

How to Use

Test Phase Results:

• The model was evalauted on Test Set of 14181 examples of different topics, the distrubtion of these topics are:

• The model achieved the follwoing accuracy for prediction on this test set:

Citation

@misc{modernbert,
      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, 
      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
      year={2024},
      eprint={2412.13663},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13663}, 
}