By concatenating layers from different LLMs, it can produce models with an exotic number of parameters (e.g., 9B with two 7B parameter models). These models are often referred to as "frankenmerges" or "Frankenstein models" by the community.

Many thanks to Abacaj for providing the fine tuned weights that were used in the creation of this base model...thanks to KatyTheCutie for inspring me to test out this script.

This idea was brought to me by The Face of Goonery, also known as Caleb Morgan. I have him to thank if fine-tuning this model turns out to be a success

How to run inference:

python

import transformers
import torch

if __name__ == "__main__":
  model_name = "Replete-AI/Phi-Elothir"
  tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
  
  model = (
      transformers.AutoModelForCausalLM.from_pretrained(
          model_name,
      )
      .to("cuda:0")
      .eval()
  )
  
  messages = [
      {"role": "user", "content": "Hello, who are you?"}
  ]
  inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
  input_ids_cutoff = inputs.size(dim=1)
  
  with torch.no_grad():
      generated_ids = model.generate(
          input_ids=inputs,
          use_cache=True,
          max_new_tokens=512,
          temperature=0.2,
          top_p=0.95,
          do_sample=True,
          eos_token_id=tokenizer.eos_token_id,
          pad_token_id=tokenizer.pad_token_id,
      )
  
  completion = tokenizer.decode(
      generated_ids[0][input_ids_cutoff:],
      skip_special_tokens=True,
  )
  
  print(completion)

Chat template

The model uses the same chat template as found in Mistral instruct models:

Join the Replete AI Discord here!

The Sauce:

yml

dtype: float16
merge_method: passthrough
slices:
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [0,2]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [1,3]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [2,4]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [3,5]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [4,6]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [5,7]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [6,8]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [7,9]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [8,10]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [9,11]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [10,12]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [11,13]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [12,14]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [13,15]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [14,16]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [15,17]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [16,18]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [17,19]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [18,20]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [19,21]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [20,22]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [21,23]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [22,24]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [23,25]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [24,26]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [25,27]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [26,28]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [27,29]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [28,30]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [29,31]
  - sources:
    - model: abacaj/phi-2-super
      layer_range: [30,32]

RichardErkhov/Replete-AI_-_Phi-Elothir-4bits

Original model description:

You are welcome here, traveler.

Named after the method used to create it, interleaving the layers of its predecessor to become far larger, giving it much more potential.

This idea was brought to me by The Face of Goonery, also known as Caleb Morgan. I have him to thank if fine-tuning this model turns out to be a success

How to run inference:

Chat template

Join the Replete AI Discord here!

The Sauce:

RichardErkhov/
Replete-AI_-_Phi-Elothir-4bits