Fine-Tuned mt5 small Model
Repository Name: sprab4/mt5_fine_tuned_model
Model Description
This repository contains the fine-tuned version of the mT5-Small model, designed for multilingual text-to-text generation tasks. The mT5 architecture supports over 100 languages, including Haitian Creole, making it suitable for text generation, summarization, and instruction-following tasks. Hence, it was fine-tuned for generating text in Haitian Creole by giving prompts in Haitian Creole.
Training Details
- Base Model: google/mt5-small
- Dataset: Haitian Creole dataset from Hugging Face which follows the alpaca format of instructions, input, and output.
- Fine-Tuning Process:
- Trained for 2 epochs using the Hugging Face Trainer.
- Validation set was used to evaluate performance periodically during fine-tuning.
- Instruction-response formats (Alpaca-style) were prioritized in the training dataset.
- Hyperparameters:
- learning_rate=1e-4
- per_device_train_batch_size=4
- gradient_accumulation_steps=4
- num_train_epochs=2
- fp16=False
- save_steps=500
- save_total_limit=2
- logging_steps=10
- optim="adamw_hf"
- weight_decay=0.05
- warmup_steps=100
Results
Quantitative Metrics
BERTScore
Varies based on the response for each prompt
BERTScore for the below prompt:
Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz
English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences
- BERTScore Precision: 0.6636
- BERTScore Recall: 0.6542
- BERTScore F1: 0.6589
Example Output
Prompt: Solèy la ap kouche ak yon bèl solèy kouche vizib nan orizon an, dekri li nan de fraz
English Translation: The sun is setting and a beautiful sunset is visible on the horizon, describe it in two sentences
Prediction: ak yon lòd efè yo fè
English Translation: and an order of effect to do
