This model is a fine-tuned version of google/flan-t5-base using PEFT (LoRA). It is designed to generate abstractive summaries in three distinct styles: Harsh (concise), Balanced (standard), and Detailed (comprehensive).

Model Details

Model Description

Model type: Sequence-to-Sequence Transformer (T5)
Language(s): English
License: MIT
Finetuned from model: google/flan-t5-base
Training Method: PEFT (LoRA)

Model Sources

Repository: Flatten
Base Model: google/flan-t5-base

Uses

Direct Use

The model interprets a prefixed prompt to determine the style of the summary.

Harsh: Generates very short, punchy summaries (approx. 35% of input length).
Balanced: Generates standard news summaries (approx. 50% of input length).
Detailed: Generates in-depth summaries (approx. 70% of input length).

Prompt Format

The input text should be prefixed with the desired style:

Summarize {Style}: {Input Text}

Example: Summarize Harsh: The Walt Disney Co. announced...

Training Details

Training Data

The model was trained on a combined dataset of 12,000 samples, split into 80% Train, 10% Validation, and 10% Test.

Style	Source Dataset	Size
Harsh	XSum	4000
Balanced	CNN/DailyMail	4000
Detailed	Multi-News	4000

Training Procedure

Training Hyperparameters

Learning Rate: 5e-4
Batch Size: 4 per device
Gradient Accumulation Steps: 2
Num Epochs: 5
Optimizer: AdamW
LR Scheduler: Linear with warmup (ratio 0.05)
Mixed Precision: BF16

LoRA Configuration

r: 32
lora_alpha: 64
lora_dropout: 0.05
target_modules: ["q", "k", "v", "o"]
bias: "none"
task_type: "SEQ_2_SEQ_LM"

Evaluation Results

Evaluated on the held-out test set (1,200 samples) at Step 6000.

Metric	Score
ROUGE-1	0.3925
ROUGE-2	0.1608
ROUGE-L	0.2776
Validation Loss	0.7824

Environmental Impact

Hardware Type: CUDA-enabled GPU
Compute: LoRA fine-tuning (Parameters: 7M trainable / 254M total)

Framework Versions

Datasets==3.6.0
Pytorch>=2.5.1
Transformers>=4.36.0
PEFT>=0.8.0

lityops/Abstractive-Style-Summarizer

Abstractive Style Summarizer