from transformers import pipeline, AutoTokenizer

MODEL_ID = "Synexian/Nexian-finetuned-llama3"

tok = AutoTokenizer.from_pretrained(MODEL_ID, use_fast=True)
pipe = pipeline(
    "text-generation",
    model=MODEL_ID,
    tokenizer=tok,
    torch_dtype="auto",
    device_map="auto",   # will use cuda:0 if available
)

def chat(user_text, system_text="You are a helpful, concise assistant.", **gen_kwargs):
    messages = [
        {"role": "system", "content": system_text},
        {"role": "user", "content": user_text},
    ]
    try:
        prompt = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
    except Exception:
        prompt = f"{system_text}\n\nUser: {user_text}\nAssistant:"
    out = pipe(
        prompt,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        return_full_text=False,
        eos_token_id=tok.eos_token_id,
        pad_token_id=tok.eos_token_id,
        **gen_kwargs
    )[0]["generated_text"]
    return out.strip()

if __name__ == "__main__":
    print(chat("Summarize gradient descent in 5 bullet points."))

Training procedure

This model was trained with SFT.

Framework versions

PEFT 0.17.0
TRL: 0.21.0
Transformers: 4.55.0
Pytorch: 2.7.1+cu128
Datasets: 3.6.0
Tokenizers: 0.21.4

Citations

Cite TRL as:

bibtex

@misc{vonwerra2022trl,
	title        = {{TRL: Transformer Reinforcement Learning}},
	author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
	year         = 2020,
	journal      = {GitHub repository},
	publisher    = {GitHub},
	howpublished = {\url{https://github.com/huggingface/trl}}
}