Quantization made by Richard Erkhov.
Marcoroni-7b-DPO-Merge is a merge of the following models using
mergekit and inspired by
Maxime Labonne's work:
1models:
2 - model: madatnlp/marcoroni-7b-v3-safetensor
3 # no parameters necessary for base model
4 - model: fblgit/UNA-TheBeagle-7b-v1
5 parameters:
6 density: 0.3
7 weight: 0.5
8 - model: udkai/Turdus
9 parameters:
10 density: 0.7
11 weight: 0.3
12merge_method: ties
13base_model: madatnlp/marcoroni-7b-v3-safetensor
14parameters:
15 normalize: true
16dtype: float16
1from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
2
3model_name_or_path = "nfaheem/Marcoroni-7b-DPO-Merge"
4model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
5 device_map="auto",
6 revision="main")
7
8tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
9
10prompt = "Write a story about llamas"
11system_message = "You are a story writing assistant"
12prompt_template=f'''{prompt}
13'''
14
15print("\n\n*** Generate:")
16
17input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
18output = model.generate(inputs=input_ids, temperature=0.7, do_sample=True, top_p=0.95, top_k=40, max_new_tokens=512)
19print(tokenizer.decode(output[0]))
20
21# Inference can also be done using transformers' pipeline
22
23print("*** Pipeline:")
24pipe = pipeline(
25 "text-generation",
26 model=model,
27 tokenizer=tokenizer,
28 max_new_tokens=512,
29 do_sample=True,
30 temperature=0.7,
31 top_p=0.95,
32 top_k=40,
33 repetition_penalty=1.1
34)
35
36print(pipe(prompt_template)[0]['generated_text'])
It's Ranked # 1 on HuggingFace Leaderboard among around 13B parameters (01/15/2024)
