This is a SOLAR-like model upscaled to 18B.
It is a frankenmerge model created using mergekit, alternating layers of Nous-Hermes-2-SOLAR-10.7B and SOLAR-10.7B-Instruct.
This model has very good writing capabilities (compared to SOLAR-10.7B), specially for role-playing.
This model was merged using the passthrough merge method.
1 slices :
2 - sources :
3 - model : NousResearch/Nous - Hermes - 2 - SOLAR - 10.7B
4 layer_range : [ 0 , 12 ]
5 - sources :
6 - model : upstage/SOLAR - 10.7B - Instruct - v1.0
7 layer_range : [ 6 , 18 ]
8 - sources :
9 - model : NousResearch/Nous - Hermes - 2 - SOLAR - 10.7B
10 layer_range : [ 13 , 25 ]
11 - sources :
12 - model : upstage/SOLAR - 10.7B - Instruct - v1.0
13 layer_range : [ 19 , 31 ]
14 - sources :
15 - model : NousResearch/Nous - Hermes - 2 - SOLAR - 10.7B
16 layer_range : [ 26 , 38 ]
17 - sources :
18 - model : upstage/SOLAR - 10.7B - Instruct - v1.0
19 layer_range : [ 32 , 44 ]
20 - sources :
21 - model : NousResearch/Nous - Hermes - 2 - SOLAR - 10.7B
22 layer_range : [ 39 , 48 ]
23
24 merge_method : passthrough
25 dtype : float16
26
tokenizer = AutoTokenizer.from_pretrained("vicgalle/franken-SOLAR-18B-v1.0")
model = AutoModelForCausalLM.from_pretrained("vicgalle/franken-SOLAR-18B-v1.0", torch_dtype=torch.float16, load_in_4bit=True)
conversation = [ {'role': 'system', 'content': SYSTEM_PROMPT}, {'role': 'user', 'content': USER_PROMPT} ]
prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, use_cache=True, max_new_tokens=1024, do_sample=True, temperature=0.8)
output_text = tokenizer.decode(outputs[0]) Detailed results can be found
here 
