from peft import AutoPeftModelForCausalLM
from transformers import GenerationConfig
from transformers import AutoTokenizer
import torch


dialogue = """
Alex: Hey, are you free this weekend?
Sarah: Yeah, what's up?
Alex: Want to go to that new restaurant downtown?
Sarah: The Italian one? I heard it's really good!
Alex: That's the one. How about Saturday around 7?
Sarah: Perfect! Should I make a reservation?
Alex: Good idea, I'll call them now.
"""

test_prompt = f"""
###Human: Summarize this following dialogue: {dialogue}
###Assistant: """

tokenizer = AutoTokenizer.from_pretrained("Wothmag07/mistral-finetuned-samsum")
model = AutoPeftModelForCausalLM.from_pretrained("Wothmag07/mistral-finetuned-samsum",
                                                  low_cpu_mem_usage=True,
                                                  return_dict=True,
                                                  torch_dtype=torch.bfloat16,
                                                  device_map="cuda")

inputs = tokenizer(test_prompt, return_tensors="pt").to("cuda")
output = model.generate(**inputs, do_sample=True, top_p=0.9, temperature=0.8, max_new_tokens=150)
tokenizer.decode(output[0], skip_special_tokens=True)

Training procedure

This model was trained with SFT.

Framework versions

TRL: 0.19.0
Transformers: 4.52.4
Pytorch: 2.6.0+cu124
Datasets: 3.6.0
Tokenizers: 0.21.1

Citations

Cite TRL as:

bibtex

@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}