SentenceTransformer Fine-Tuned for Amharic Retrieval

Model Details

• Model Type: Sentence Transformer
• Base Model: sentence-transformers/paraphrase-xlm-r-multilingual-v1
• Training Task: Triplet loss with Matryoshka loss
• Language: Amharic
• Maximum Sequence Length: 128 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity

Training Overview

• Training Data: Custom Amharic QA triplets (with positive and negative examples)
• Training Strategy:
  The model was finetuned using a combination of triplet loss and a Matryoshka loss, with evaluation performed using a TripletEvaluator.
• Hyperparameters:
  • Epochs: 3
  • Batch Size: 16
  • Learning Rate: 1e-6
  • Warmup Ratio: 0.08
  • Weight Decay: 0.05

Evaluation

Usage

1. Install Sentence Transformers:
   pip install -U sentence-transformers
2. Load the Model:
   python

   1from sentence_transformers import SentenceTransformer 2 3model = SentenceTransformer("abdulmunimjemal/xlm-r-retrieval-am-v5") 4sentences = [ 5 "ሰማይ ምን አይነት ቀለም ነው?", 6 "ሰማይ ሰማያዊ ቀለም አለው።" , 7 "እኔ ምሳ እንጀራ በላሁ።" , 8 "ባሕር ምን አይነት ቀለም ነው?", 9 "አየር በምድር ዙሪያ ያለ ነው።" 10] 11embeddings = model.encode(sentences) 12print(embeddings.shape) # Expected output: (5, 768)
3. Compute Similarity:
   python

   1from sklearn.metrics.pairwise import cosine_similarity 2similarities = cosine_similarity(embeddings, embeddings) 3print(similarities.shape) # Expected output: (5, 5)

Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: XLMRobertaModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_mean_tokens': True, ...})
)

Training Environment

• Python: 3.11.11
• Sentence Transformers: 3.3.1
• Transformers: 4.47.1
• PyTorch: 2.5.1+cu124
• Accelerate: 1.2.1
• Datasets: 3.2.0
• Tokenizers: 0.21.0

Citation