bge-m3 fine-tuned for Quran thematic retrieval (Wahi)

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • wahi-quran-retrieval-pairs
• Languages: en, fi, ar
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'embedding_dimension': 1024, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Information Retrieval

• Dataset: quranqa-dev
• Evaluated with InformationRetrievalEvaluator

Training Details

Training Dataset

wahi-quran-retrieval-pairs

• Dataset: wahi-quran-retrieval-pairs
• Size: 11,218 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 7 tokens mean: 9.1 tokens max: 12 tokens	min: 52 tokens mean: 171.12 tokens max: 256 tokens

• Samples:

  anchor	positive
  من هم قوم شعيب؟	And to [the people of] Madyan [We sent] their brother Shu'ayb. He said, "O my people, worship Allah; you have no deity other than Him. There has come to you clear evidence from your Lord. So fulfill the measure and weight and do not deprive people of their due and cause not corruption upon the earth after its reformation. That is better for you, if you should be believers And do not sit on every path, threatening and averting from the way of Allah those who believe in Him, seeking to make it [seem] deviant. And remember when you were few and He increased you. And see how was the end of the corrupters And if there should be a group among you who has believed in that with which I have been sent and a group that has not believed, then be patient until Allah judges between us. And He is the best of judges Said the eminent ones who were arrogant among his people, "We will surely evict you, O Shu'ayb, and those who have believed with you from our city, or you must return to our religion." He...
  من هم قوم شعيب؟	وإلى مدين أخاهم شعيبا قال يا قوم اعبدوا الله ما لكم من إله غيره قد جاءتكم بينة من ربكم فأوفوا الكيل والميزان ولا تبخسوا الناس أشياءهم ولا تفسدوا في الأرض بعد إصلاحها ذلكم خير لكم إن كنتم مؤمنين. ولا تقعدوا بكل صراط توعدون وتصدون عن سبيل الله من آمن به وتبغونها عوجا واذكروا إذ كنتم قليلا فكثركم وانظروا كيف كان عاقبة المفسدين. وإن كان طائفة منكم آمنوا بالذي أرسلت به وطائفة لم يؤمنوا فاصبروا حتى يحكم الله بيننا وهو خير الحاكمين. قال الملأ الذين استكبروا من قومه لنخرجنك يا شعيب والذين آمنوا معك من قريتنا أو لتعودن في ملتنا قال أولو كنا كارهين. قد افترينا على الله كذبا إن عدنا في ملتكم بعد إذ نجانا الله منها وما يكون لنا أن نعود فيها إلا أن يشاء الله ربنا وسع ربنا كل شيء علما على الله توكلنا ربنا افتح بيننا وبين قومنا بالحق وأنت خير الفاتحين. وقال الملأ الذين كفروا من قومه لئن اتبعتم شعيبا إنكم إذا لخاسرون. فأخذتهم الرجفة فأصبحوا في دارهم جاثمين. الذين كذبوا شعيبا كأن لم يغنوا فيها الذين كذبوا شعيبا كانوا هم الخاسرين. فتولى عنهم وقال يا قوم لقد أبلغتكم رسالات ربي ونصحت لكم فكيف آسى على قوم ...
  Who are the people of Shu'ayb?	And to [the people of] Madyan [We sent] their brother Shu'ayb. He said, "O my people, worship Allah; you have no deity other than Him. There has come to you clear evidence from your Lord. So fulfill the measure and weight and do not deprive people of their due and cause not corruption upon the earth after its reformation. That is better for you, if you should be believers And do not sit on every path, threatening and averting from the way of Allah those who believe in Him, seeking to make it [seem] deviant. And remember when you were few and He increased you. And see how was the end of the corrupters And if there should be a group among you who has believed in that with which I have been sent and a group that has not believed, then be patient until Allah judges between us. And He is the best of judges Said the eminent ones who were arrogant among his people, "We will surely evict you, O Shu'ayb, and those who have believed with you from our city, or you must return to our religion." He...

• Loss: MultipleNegativesRankingLoss with these parameters:
  json

  1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false, 5 "directions": [ 6 "query_to_doc" 7 ], 8 "partition_mode": "joint", 9 "hardness_mode": null, 10 "hardness_strength": 0.0 11}

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 24
• num_train_epochs: 4
• learning_rate: 1.5e-05
• warmup_steps: 0.1
• weight_decay: 0.01
• bf16: True
• per_device_eval_batch_size: 64
• push_to_hub: True
• hub_private_repo: True
• hub_model_id: wahi-quran-bge-m3-v3
• load_best_model_at_end: True
• seed: 12
• batch_sampler: no_duplicates

All Hyperparameters

Training Logs

Training Time

• Training: 8.1 minutes
• Evaluation: 11.1 seconds
• Total: 8.3 minutes

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.5.1
• Transformers: 5.9.0
• PyTorch: 2.12.0+cu130
• Accelerate: 1.13.0
• Datasets: 4.8.5
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

MultipleNegativesRankingLoss