import torch
from transformers import LlamaForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = LlamaForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B-Instruct",
    dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model,
    "gabe-zhang/Llama-PaperSummarization-LoRA"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B-Instruct")

Training Details

Parameter	Value
Base Model	Llama-3.2-1B-Instruct (1.3GB)
LoRA Rank	8
Target Modules	q_proj, v_proj
Trainable Parameters	~850K (0.07%)
Context Length	10,182 tokens
Gradient Accumulation	4 steps
Training Steps	5,000
Evaluation Interval	Every 20 steps
Training Time	~28 hours on RTX A6000

Dataset

Fine-tuned on 10% of ccdv/arxiv-summarization:

Split	Samples	Avg. Article Tokens	Avg. Abstract Tokens
Train	~20,000	6,038	299
Validation	~640	5,894	172
Test	6,440	5,905	174

Training Code

github.com/gabe-zhang/paper2summary

License

Llama 3.2: Llama 3.2 Community License
LoRA Adapter: MIT License

Built with Llama.