Model Card for AmoooEBI/xlm-roberta-fa-qa-finetuned-on-PersianQA
This model is a version of pedramyazdipoor/persian_xlm_roberta_large, fine-tuned with LoRA for extractive question answering on the Persian language using the PersianQA dataset.
Model Details
Model Description
This is an XLM-RoBERTa model fine-tuned on the SajjadAyoubi/persian_qa dataset for extractive question answering in Persian. The model was trained using the parameter-efficient LoRA method, which significantly speeds up training while achieving high performance. It is designed to extract an answer to a question directly from a given context. The fine-tuning process has made it a top-performing model for this task in Persian.
- Developed by: Amir Mohammad Ebrahiminasab
- Shared by: Amir Mohammad Ebrahiminasab
- Model type: xlm-roberta
- Language(s): fa (Persian)
- License: MIT
- Finetuned from model:
pedramyazdipoor/persian_xlm_roberta_large
Model Sources
Uses
Direct Use
The model can be used directly for extractive question answering in Persian using the pipeline function.
1from transformers import pipeline
2
3qa_pipeline = pipeline(
4 "question-answering",
5 model="AmoooEBI/xlm-roberta-fa-qa-finetuned-on-PersianQA",
6 tokenizer="AmoooEBI/xlm-roberta-fa-qa-finetuned-on-PersianQA"
7)
8
9context = "مهانداس کارامچاند گاندی رهبر سیاسی و معنوی هندیها بود که ملت هند را در راه آزادی از استعمار امپراتوری بریتانیا رهبری کرد."
10question = "گاندی که بود؟"
11
12result = qa_pipeline(question=question, context=context)
13print(f"Answer: '{result['answer']}'")
Bias, Risks, and Limitations
The model's performance is highly dependent on the quality and domain of the context provided. It was trained on the PersianQA dataset, which is largely based on Wikipedia articles. Therefore, its performance may degrade on texts with different styles, such as conversational or technical documents.
Like ParsBERT, this model also shows a preference for shorter answers, with its Exact Match score dropping for answers longer than the dataset's average. However, its F1-score remains high, indicating it can still identify the correct span of text with high token overlap.
Recommendations
Users should be aware of the model's limitations, particularly the potential for lower Exact Match scores on long-form answers. For applications requiring high precision, outputs should be validated.
How to Get Started with the Model
Use the code below to get started with the model using PyTorch.
1from transformers import AutoTokenizer, AutoModelForQuestionAnswering
2import torch
3
4tokenizer = AutoTokenizer.from_pretrained("AmoooEBI/xlm-roberta-fa-qa-finetuned-on-PersianQA")
5model = AutoModelForQuestionAnswering.from_pretrained("AmoooEBI/xlm-roberta-fa-qa-finetuned-on-PersianQA")
6
7context = "پایتخت اسپانیا شهر مادرید است."
8question = "پایتخت اسپانیا کجاست؟"
9
10inputs = tokenizer(question, context, return_tensors="pt")
11with torch.no_grad():
12 outputs = model(**inputs)
13
14answer_start_index = outputs.start_logits.argmax()
15answer_end_index = outputs.end_logits.argmax()
16
17predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
18answer = tokenizer.decode(predict_answer_tokens)
19
20print(f"Question: {question}")
21print(f"Answer: {answer}")
Training Details
Training Data
The model was fine-tuned on the SajjadAyoubi/persian_qa dataset, which contains question-context-answer triplets in Persian, primarily from Wikipedia.
Training Procedure
Preprocessing
The training data was tokenized using the XLM-RoBERTa tokenizer. Contexts longer than the model's maximum input size were split into overlapping chunks using a sliding window (doc_stride=128). The start and end positions of the answer token were then mapped to these chunks.
Training Hyperparameters
- Training regime: LoRA (Parameter-Efficient Fine-Tuning)
r: 16lora_alpha: 32lora_dropout: 0.1target_modules: ["query", "value"]- Learning Rate: 2 × 10⁻⁵
- Epochs: 8
- Batch Size: 8
Speeds, Sizes, Times
- Training Time: ~3 hours on a single GPU
- Trainable Parameters: 0.281% of model parameters
Evaluation
Testing Data, Factors & Metrics
Testing Data
The evaluation was performed on the validation set of the SajjadAyoubi/persian_qa dataset.
Factors
- Answer Presence: Questions with and without answers
- Answer Length: Shorter vs. longer than the average (22.78 characters)
Metrics
- F1-Score: Measures token overlap
- Exact Match (EM): Measures perfect span match
Results
Overall Performance on the Validation Set (LoRA Fine-Tuned)
| Model Status | Exact Match | F1-Score |
|---|
| Fine-Tuned Model (LoRA) | 69.90% | 84.85% |
Performance on Data Subsets
| Case Type | Exact Match | F1-Score |
|---|
| Has Answer | 62.06% | 83.42% |
| No Answer | 88.17% | 88.17% |
| Answer Length | Exact Match | F1-Score |
|---|
| Longer than Avg. | 49.22% | 81.88% |
| Shorter than Avg. | 62.95% | 80.20% |
Environmental Impact
- Hardware Type: T4 GPU
- Training Time: ~3 hours
- Cloud Provider: Google Colab
- Carbon Emitted: Not calculated
Technical Specifications
Model Architecture and Objective
The model uses the XLM-RoBERTa-Large architecture with a question-answering head. The training objective was to minimize the loss for the start and end token classification.
Compute Infrastructure
- Hardware: Single NVIDIA T4 GPU
- Software:
transformers, torch, datasets, evaluate, peft
Model Card Authors
Amir Mohammad Ebrahiminasab
Model Card Contact
