SentenceTransformer based on FacebookAI/xlm-roberta-base

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: FacebookAI/xlm-roberta-base
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • en-sa

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Knowledge Distillation

• Dataset: en-sa
• Evaluated with MSEEvaluator

Translation

• Dataset: en-sa
• Evaluated with TranslationEvaluator

Training Details

Training Dataset

en-sa

• Dataset: en-sa
• Size: 257,886 training samples
• Columns: english, non_english, and label
• Approximate statistics based on the first 1000 samples:

  	english	non_english	label
  type	string	string	list
  details	min: 12 tokens mean: 34.23 tokens max: 113 tokens	min: 14 tokens mean: 49.72 tokens max: 128 tokens	size: 768 elements

• Samples:

  english	non_english	label
  There was no Mughal tradition of primogeniture, the systematic passing of rule, upon an emperor's death, to his eldest son.[object Object]	चक्रवर्तिनः मृत्योः अनन्तरं तस्य शासनस्य व्यवस्थितरूपेण सङ्क्रमणस्य, मुघलपरम्परायाः ज्येष्ठपुत्राधिकारपद्धतिः नासीत्।[object Object]	[-0.5880301594734192, -0.20026817917823792, 0.372330904006958, -0.9807565808296204, -0.35607191920280457, ...]
  The four sons of Shah Jahan all held governorships during their father's reign.[object Object]	शाह्-जहाँ-नामकस्य चत्वारः पुत्राः, सर्वे पितुः शासनकाले शासकपदम् अधारयन्।[object Object]	[-0.5090229511260986, 0.33517003059387207, 0.27507224678993225, -0.05707915127277374, -0.5126022100448608, ...]
  In this regard he discusses the correlation between social opportunities of education and health and how both of these complement economic and political freedoms as a healthy and well-educated person is better suited to make informed economic decisions and be involved in fruitful political demonstrations etc.[object Object]	अस्मिन् विषये सः शिक्षणस्य स्वास्थ्यस्य च सामाजिकावकाशानाम् अन्योन्य-सम्बन्धस्य, तथा च एतद्द्वयम् अपि आर्थिक-राजनैतिक-स्वातन्त्र्ययोः कथं पूरकं भवतः इति च चर्चां करोति, यतोहि स्वस्था सुशिक्षिता च व्यक्तिः ज्ञानपूर्वम् आर्थिकविषयान् निर्णेतुं तथा फलप्रदेषु राजनैतिकेषु प्रतिपादनादिषु संलग्नः भवितुं च अधिकारी भवति इति।[object Object]	[0.16507332026958466, -0.1722974181175232, 0.02585001103579998, 0.36087149381637573, -0.6401643753051758, ...]

• Loss: MSELoss

Evaluation Dataset

en-sa

• Dataset: en-sa
• Size: 1,000 evaluation samples
• Columns: english, non_english, and label
• Approximate statistics based on the first 1000 samples:

  	english	non_english	label
  type	string	string	list
  details	min: 4 tokens mean: 21.38 tokens max: 68 tokens	min: 4 tokens mean: 27.89 tokens max: 91 tokens	size: 768 elements

• Samples:

  english	non_english	label
  """So they cast him out of the vineyard, and killed him. What therefore shall the lord of the vineyard do unto them?"""	ततस्ते तं क्षेत्राद् बहि र्निपात्य जघ्नुस्तस्मात् स क्षेत्रपतिस्तान् प्रति किं करिष्यति?	[-0.06878167390823364, -0.5150429606437683, -0.09011576324701309, -0.7458725571632385, 0.050420328974723816, ...]
  Avogadro application window opens.	Avogadro एप्लिकेशन् विण्डो उद्घट्यते ।	[0.9054689407348633, -0.2203768789768219, -0.19827595353126526, 0.23870715498924255, -0.3162331283092499, ...]
  Svangah: One whose limbs are beautiful.	स्वंग:यस्य अङ्गानि सुन्दराणि सन्ति	[0.6443825960159302, 0.4850354492664337, -0.4563218355178833, -0.4771449863910675, 0.6588209867477417, ...]

• Loss: MSELoss

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• learning_rate: 2e-05
• num_train_epochs: 10
• warmup_ratio: 0.1

All Hyperparameters

Training Logs

Framework Versions

• Python: 3.10.17
• Sentence Transformers: 4.1.0
• Transformers: 4.46.3
• PyTorch: 2.2.0+cu121
• Accelerate: 1.1.1
• Datasets: 2.18.0
• Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

MSELoss