T5-Adapter-Financial: AdaLoRA Fine-Tuned T5 Model for Financial News Summarization

Model Overview

• Model Architecture: T5-base
• Task: Text summarization (financial news → news title)
• Fine-Tuning Technique: AdaLoRA (Adaptive Low-Rank Adaptation)
• Pretrained Checkpoint: mrm8488/t5-base-finetuned-summarize-news
• Dataset: ashraq/financial-news-articles (80/20 train/test split)
• Evaluation Metric: ROUGE
• Framework: Hugging Face Transformers + PEFT + PyTorch

AdaLoRA Configuration

• Initial Rank (init_r): 6
• Rank Adaptation: tinit=100, tfinal=1000, deltaT=10
• Alpha (lora_alpha): 64
• Target Modules: q, v (query and value projections in attention)
• Dropout (lora_dropout): 0.05
• Orthogonal Regularization Weight: 0.5
• Task Type: Sequence-to-sequence LM

Only AdaLoRA parameters are trained; the base T5 parameters remain frozen. The rank is adapted dynamically during training.

Evaluation Results

These results were obtained after 15 epochs of AdaLoRA fine-tuning on 4 GPUs.