This project involves training and fine-tuning a language model on a Basque language dataset using LoRA (Low-Rank Adaptation) and performing evaluations to measure its effectiveness. The workflow is split into two key phases:
Training and Fine-Tuning
Evaluation Phase
The purpose of this README is to document the steps, hyperparameters, and methodologies used in these processes and provide a clear understanding of the project's structure.
This notebook contains the implementation for training and fine-tuning a language model on a Basque dataset using LoRA. The objective is to adapt a large pre-trained model to perform better on Basque-specific tasks.
Key Steps:
Model Initialization:
Utilizes the FastLanguageModel to load the base model.
Applies LoRA for parameter-efficient fine-tuning.
Hyperparameters:
Final Hyperparameters:
r=32: Higher rank allows capturing more complex features, improving memory and performance.
lora_alpha=16: Balances the impact of LoRA layers.
lora_dropout=0: Ensures no dropout during fine-tuning.
target_modules: Includes layers like q_proj, k_proj, and others for adaptation.
use_rslora=True: Applies the Rank-Stabilized LoRA method for better scaling.
use_gradient_checkpointing="unsloth": Reduces memory usage for large models.
Previously Tried Hyperparameters:
r=8: Limited complexity, resulting in underperformance.
Other settings remained consistent.
Training Process:
Implements standard optimizers and learning rate schedules.
Monitors loss and performance during each epoch.
Challenges and Solutions:
Challenge: Underfitting with r=8.
Solution: Increased r to 32, which enhanced model adaptability.
2. Evaluation Phase
File: Run_Local_Model_6604.ipynb
This notebook evaluates the fine-tuned model's performance using metrics such as accuracy, precision, recall, and F1 score.
Key Steps:
Model Loading:
Loads the fine-tuned model.
Ensures all necessary pre-trained weights and configurations are correctly initialized.
Evaluation Metrics:
Accuracy: Measures the proportion of correct predictions.
Precision: Assesses how many predicted positives are true positives.
Recall: Determines how many actual positives were identified.
F1 Score: Balances precision and recall for a holistic performance view.
Results:
The evaluation metrics indicated substantial improvement after fine-tuning.
Demonstrates the model’s capability to generalize on unseen Basque language tasks.
Challenges and Observations:
Challenge: Initial evaluations showed low F1 scores with under-optimized settings.
Solution: Fine-tuning hyperparameters improved all metrics significantly.
