mT5-LatinSummarizerModel: Fine-Tuned Model for Latin NLP

Overview

• Translate between English and Latin.
• Summarize Latin texts effectively.
• Leverage extractive and abstractive summarization techniques.
• Utilize curriculum learning for improved training.

Installation & Usage

Project Structure

.
├── final_pipeline (Trained for 30 light epochs with optimizations, and then finetuned on 100 on the small HQ summaries dataset)
│   ├── no_stanza
│   ├── with_stanza
├── initial_pipeline (Trained for 6 epochs without optimizations)
│   ├── mt5-small-en-la-translation-epoch5
├── install_large_models.sh
└── README.md

Training Methodology

1. Initial Training Pipeline (6 epochs): Used the full dataset without optimizations.
2. Final Training Pipeline (30 light epochs): Used 10% of training data per epoch for efficiency.
3. Fine-Tuning (100 epochs): Focused on the 4750 high-quality summaries for final optimization.

Training Configurations:

• Hardware: 16GB VRAM GPU (lab machines via SSH).
• Batch Size: Adaptive due to GPU memory constraints.
• Gradient Accumulation: Enabled for larger effective batch sizes.
• LoRA-based fine-tuning: LoRA Rank 8, Scaling Factor 32.
• Dynamic Sequence Length Adjustment: Increased progressively.
• Learning Rate: 5 × 10^-4 with warm-up steps.
• Checkpointing: Frequent saves to mitigate power outages.

Evaluation & Results

• chrF Score (en→la): 33.60 (with Stanza tags) vs 18.03 BLEU (without Stanza).
• Summarization Density: Maintained at ~6%.

Observations:

• Pre-training on extractive summaries was crucial.
• The model retained some excessive extraction, indicating room for further improvement.

License

Citation