Georgian Language Question-Answering Model
Project Overview
This project fine-tunes a pre-trained GPT-2 model for a Georgian language question-answering task. The model generates answers to questions in Georgian using a dataset containing question-answer pairs. The fine-tuned GPT-2 model is evaluated using common NLP metrics such as BLEU and ROUGE. The trained model can be used for automatic question answering in Georgian.
Key Features
- Fine-tuned GPT-2 model for Georgian question-answering.
- Trained on the MrDimitri/georgian-quora-faq dataset.
- Evaluation using metrics such as BLEU and ROUGE.
- Predictions are generated for test samples and compared against ground truth answers.
Requirements
- Python 3.6+
- PyTorch 1.9+ (for GPU support)
- Hugging Face transformers and datasets library
- Google Colab (recommended for GPU access)
- SacreBLEU and Rouge-Score libraries for evaluation metrics.
Dataset
The dataset used for training is MrDimitri/georgian-quora-faq from Hugging Face. It contains a collection of question-answer pairs in Georgian, ideal for training a question-answering model.
Dataset Structure
The dataset consists of two columns:
- question: The input question.
- answer: The expected answer.
Model
The model architecture is based on GPT-2, a Causal Language Model. It is a transformer-based model that is fine-tuned for the task of question answering by generating answers in the Georgian language.
Model Details
- Pre-trained Model: gpt2 (GPT-2 for general language modeling).
- Tokenizer: The model uses the GPT-2 tokenizer to preprocess input text.
- Special Tokens:
pad_token_id: Padding token ID to handle sequences of different lengths.eos_token_id: End-of-sequence token used for sequence generation.
Training Process
Data Preprocessing
- Tokenization of input (question) and output (answer) sequences.
- Padding and truncation to a fixed length for model input.
Model Fine-Tuning
- The model is trained for 100 epochs.
- A batch size of 16 for both training and evaluation.
- Loss function: Cross-entropy loss for sequence generation.
Training Arguments
- evaluation_strategy: Evaluates the model at the end of each epoch.
- num_train_epochs: Number of training epochs (100).
- save_strategy: Saves the model at the end of each epoch.
- learning_rate: Set to 5e-5.
- gradient_accumulation_steps: Set to 2 for memory efficiency.
Evaluation
Metrics
BLEU Score
A metric for evaluating text generation in machine translation and question answering tasks.
ROUGE Scores
- ROUGE-1: Measures the overlap of unigrams.
- ROUGE-2: Measures the overlap of bigrams.
- ROUGE-L: Measures the longest common subsequence overlap.
