You can deploy this model and start earning money today!

mariam8/
tinyllama-xsum-lora

Name: tinyllama-xsum-lora
Availability: InStock

transformers

safetensors

endpoints_compatible

Views

No views yet

Model Card for Model ID

license: apache-2.0 base_model: TinyLlama/TinyLlama-1.1B-Chat-v1.0 library_name: peft tags:

lora
peft
summarization
text-generation
tinyllama datasets:
EdinburghNLP/xsum language:
en pipeline_tag: summarization

TinyLlama-1.1B LoRA — News Summarization (XSum)

This is a LoRA (PEFT) adapter fine-tuned on top of TinyLlama/TinyLlama-1.1B-Chat-v1.0 for single-sentence news summarization, trained on a subset of the XSum dataset.

This is an adapter only — it must be loaded on top of the base model (see Usage below).

Model Details

Base model: TinyLlama-1.1B-Chat-v1.0
Fine-tuning method: LoRA (Low-Rank Adaptation) via PEFT
Task: Abstractive summarization (news articles → one-sentence summary)
Training data: 3,000 examples sampled from XSum train split
Trainable parameters: <1% of total model parameters (LoRA rank=16, alpha=32, targeting q/k/v/o attention projections)
Quantization: 4-bit (NF4) via bitsandbytes during training

Training Procedure

Epochs: 3
Batch size: 4 (effective batch size 16 with gradient accumulation)
Learning rate: 2e-4
LoRA config: r=16, alpha=32, dropout=0.05, target_modules=["q_proj", "v_proj", "k_proj", "o_proj"]
Hardware: Single T4 GPU (Google Colab, free tier)

Evaluation

Evaluated on a held-out sample of 30 examples from the XSum validation split, comparing the base model (zero-shot) against the LoRA fine-tuned model.

Metric	Before (base, zero-shot)	After (LoRA fine-tuned)	Improvement
ROUGE-1	0.138	0.238	+72%
ROUGE-2	—	0.096	—
ROUGE-L	—	0.184	—
ROUGE-Lsum	—	0.191	—

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

base_model_name = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
adapter_name = "your-username/tinyllama-xsum-lora"  # replace with your repo id

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    torch_dtype=torch.float16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, adapter_name)

prompt = (
    "<|system|>\nYou are a helpful assistant that summarizes news articles "
    "in one short sentence.</s>\n"
    "<|user|>\nSummarize the following article:\n{your_article_here}</s>\n"
    "<|assistant|>\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=60, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

Trained on a small subset (3,000 examples) of XSum for a short training run — not intended to match dedicated summarization models (e.g. BART, PEGASUS) in quality.
Base model (TinyLlama-1.1B) is small; summaries may occasionally be generic or miss key details from longer articles.
Evaluated on a small sample (30 examples); ROUGE scores are indicative, not a rigorous benchmark.

Intended Use

Educational / portfolio project demonstrating parameter-efficient fine-tuning (LoRA/PEFT) of a small LLM for a summarization task under limited compute (free-tier Colab GPU).

mariam8/tinyllama-xsum-lora