You can deploy this model and start earning money today!

RichardErkhov/
Replete-AI_-_Phi-Delthanar-gguf

Name: Replete-AI_-_Phi-Delthanar-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Phi-Delthanar - GGUF

Model creator: https://huggingface.co/Replete-AI/
Original model: https://huggingface.co/Replete-AI/Phi-Delthanar/

Name	Quant method	Size
Phi-Delthanar.Q2_K.gguf	Q2_K	2.82GB
Phi-Delthanar.IQ3_XS.gguf	IQ3_XS	3.05GB
Phi-Delthanar.IQ3_S.gguf	IQ3_S	3.19GB
Phi-Delthanar.Q3_K_S.gguf	Q3_K_S	3.19GB
Phi-Delthanar.IQ3_M.gguf	IQ3_M	3.38GB
Phi-Delthanar.Q3_K.gguf	Q3_K	3.67GB
Phi-Delthanar.Q3_K_M.gguf	Q3_K_M	3.67GB
Phi-Delthanar.Q3_K_L.gguf	Q3_K_L	4.09GB
Phi-Delthanar.IQ4_XS.gguf	IQ4_XS	3.96GB
Phi-Delthanar.Q4_0.gguf	Q4_0	4.14GB
Phi-Delthanar.IQ4_NL.gguf	IQ4_NL	4.17GB
Phi-Delthanar.Q4_K_S.gguf	Q4_K_S	4.18GB
Phi-Delthanar.Q4_K.gguf	Q4_K	4.51GB
Phi-Delthanar.Q4_K_M.gguf	Q4_K_M	4.51GB
Phi-Delthanar.Q4_1.gguf	Q4_1	4.58GB
Phi-Delthanar.Q5_0.gguf	Q5_0	5.03GB
Phi-Delthanar.Q5_K_S.gguf	Q5_K_S	5.03GB
Phi-Delthanar.Q5_K.gguf	Q5_K	5.22GB
Phi-Delthanar.Q5_K_M.gguf	Q5_K_M	5.22GB
Phi-Delthanar.Q5_1.gguf	Q5_1	5.48GB
Phi-Delthanar.Q6_K.gguf	Q6_K	5.98GB
Phi-Delthanar.Q8_0.gguf	Q8_0	7.74GB

Original model description:

license: mit language:

en thumbnail: "https://cdn-uploads.huggingface.co/production/uploads/6589d7e6586088fd2784a12c/iYImJKf2HZZZJ9IwDSN00.png"

The forest is with you.

Named after the method used to create it, interleaving the layers of its predecessor to become far larger, giving it much more potential.

Del'thanar is a supposed ancient treeant, and I couldn't think of a better naming convention for a model that was created using the passthrough method.

By concatenating layers from different LLMs, it can produce models with an exotic number of parameters (e.g., 9B with two 7B parameter models). These models are often referred to as "frankenmerges" or "Frankenstein models" by the community.

Many thanks to Abacaj for providing the fine tuned weights that were used in the creation of this base model. You can find the full script for how the model was merged here...thanks to KatyTheCutie for inspiring me to test out this script.

This idea was brought to me by The Face of Goonery, also known as Caleb Morgan. I have him to thank if fine-tuning this model turns out to be a success...he also helped me to make this model even larger than the prior one.

How to run inference:

python

import transformers
import torch

if __name__ == "__main__":
  model_name = "Replete-AI/Phi-Delthanar"
  tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
  
  model = (
      transformers.AutoModelForCausalLM.from_pretrained(
          model_name,
      )
      .to("cuda:0")
      .eval()
  )
  
  messages = [
      {"role": "user", "content": "Hello, who are you?"}
  ]
  inputs = tokenizer.apply_chat_template(messages, return_tensors="pt").to(model.device)
  input_ids_cutoff = inputs.size(dim=1)
  
  with torch.no_grad():
      generated_ids = model.generate(
          input_ids=inputs,
          use_cache=True,
          max_new_tokens=512,
          temperature=0.2,
          top_p=0.95,
          do_sample=True,
          eos_token_id=tokenizer.eos_token_id,
          pad_token_id=tokenizer.pad_token_id,
      )
  
  completion = tokenizer.decode(
      generated_ids[0][input_ids_cutoff:],
      skip_special_tokens=True,
  )
  
  print(completion)

Chat template

The model uses the same chat template as found in Mistral instruct models:

Join the Replete AI Discord here!

RichardErkhov/Replete-AI_-_Phi-Delthanar-gguf