You can deploy this model and start earning money today!

A7med-Ame3/
flan-t5-base-xsum-lora

Name: flan-t5-base-xsum-lora
Availability: InStock

Views

No views yet

FLAN-T5 Base XSum LoRA Adapter

This repository provides a LoRA fine-tuned adapter for
:contentReference[oaicite:1]{index=1}
trained on the
:contentReference[oaicite:2]{index=2}
for abstractive text summarization.

The adapter uses parameter-efficient fine-tuning (PEFT) to adapt the base model while keeping most of its parameters frozen.

Model Details

Base model: google/flan-t5-base
Task: Abstractive Summarization
Fine-tuning method: LoRA
Dataset: EdinburghNLP/xsum
Framework: Transformers + PEFT

Summary

Setting	Value
Base model	`google/flan-t5-base` (250M params)
Dataset	`EdinburghNLP/xsum`
LoRA rank (r)	16
LoRA alpha	32
Target modules	`q`, `v` (attention projections)
Trainable params	~0.4% of total
Epochs	3
Batch size	8 (effective 16 with grad accum)
Learning rate	3e-4
Evaluation metric	ROUGE-1 / ROUGE-2 / ROUGE-L

Training Procedure

The model was fine-tuned on the XSum dataset, which contains BBC news articles paired with a single-sentence summary.

Training used LoRA adapters applied to the attention projection layers while keeping the base model frozen.
This reduces the number of trainable parameters while maintaining strong summarization performance.

How to Use

Install dependencies:

bash

pip install transformers peft accelerate

# Load the adapter:
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")

model = PeftModel.from_pretrained(base_model, "A7med-Ame3/flan-t5-base-xsum-lora")

text = "summarize: Artificial intelligence is transforming many industries..."

inputs = tokenizer(text, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=60)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

# Next Steps

# لإry target_modules=["q", "k", "v", "o"] for potentially better ROUGE scores

# Increase LoRA rank to r=32 for more expressive adapters

# Use google/flan-t5-large for higher-quality production models

# Tune beam search parameters to improve inference quality