This adapter is intended for an academic lab assignment on Transformers, language models, and PEFT. It can be used to generate short summaries from messenger-style dialogues.

Prompt format:

text

Dialogue:
{dialogue}

Summary:

The model is expected to continue the prompt with the summary.

How to Use

python

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "openai-community/gpt2"
adapter_id = "luanacarolina/gpt2-samsum-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()
model.eval()

dialogue = """Amanda: I baked cookies. Do you want some?
Jerry: Sure!
Amanda: I'll bring you tomorrow :-)"""

prompt = f"Dialogue:\n{dialogue.strip()}\n\nSummary:\n"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=96,
        num_beams=4,
        do_sample=False,
        no_repeat_ngram_size=3,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated_ids = outputs[0][inputs["input_ids"].shape[-1]:]
summary = tokenizer.decode(generated_ids, skip_special_tokens=True).strip()
print(summary)

For the lab submission, the repository also includes a separate Python file named lora_summarizer_submission.py with:

load_model()
generate_summary(model, tokenizer, dialogue) -> str

Training Data

The adapter was fine-tuned on knkarthick/samsum, a dialogue summarization dataset with messenger-like conversations and human-written summaries.

The dataset contains approximately:

14,731 training examples
818 validation examples
819 test examples

Training Procedure

The task was formatted as causal language modeling. Each example was converted to:

text

Dialogue:
{dialogue}

Summary:
{summary}

During training, prompt tokens were masked with -100, so the loss was computed only on the summary tokens.

LoRA Configuration

LoRA rank: 8
LoRA alpha: 16
LoRA dropout: 0.05
Target modules: c_attn, c_proj, c_fc
Trainable parameters: about 1.18M
Trainable percentage: about 0.94% of GPT-2 parameters

Main Hyperparameters

Base model: openai-community/gpt2
Epochs: 1
Learning rate: 2e-4
Batch size: 2
Gradient accumulation steps: 8
Optimizer: AdamW
Max sequence length: 768
Max target length: 128
Warmup steps: 50

Evaluation

The training script evaluates on the SAMSum validation split and reports validation loss. No additional benchmark metric is provided in this repository.

For qualitative evaluation, users can compare generated summaries against SAMSum reference summaries.

Limitations

GPT-2 is a relatively small decoder-only model and was not originally designed as a summarization model.
The model may hallucinate details or repeat text, especially for long or ambiguous dialogues.
The adapter was trained for English messenger-style dialogues and may not generalize well to other languages or formal documents.
Generated summaries should be manually reviewed before being used in any important context.

Out-of-Scope Use

This adapter is not intended for high-stakes summarization, legal/medical summaries, or production systems requiring factual guarantees.

Environmental Impact

The adapter was trained locally on a single NVIDIA GeForce RTX 3060 Laptop GPU. The PEFT approach reduces the number of trainable parameters compared with full fine-tuning.

Citation

LoRA:

bibtex

@article{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  journal={arXiv preprint arXiv:2106.09685},
  year={2021}
}

SAMSum:

bibtex

@article{gliwa2019samsum,
  title={SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization},
  author={Gliwa, Bogdan and Mochol, Iwona and Biesek, Maciej and Wawer, Aleksander},
  journal={arXiv preprint arXiv:1911.12237},
  year={2019}
}