The model has been quantized as part of the project GPTStonks. It works with transformers>=4.33.0 and it can run on a consumer GPU, with less than 3GB of GPU RAM. The libraries optimum, auto-gptq, peft and accelerate should also be installed.

Here is a sample code to load the model and run inference with it using sampling as decoding strategy:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "daedalus314/Griffin-3B-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')

text = """### HUMAN:
What is artifical intelligence?

### RESPONSE:
"""
inputs = tokenizer(text, return_tensors="pt").to(0)

out = quant_model.generate(
    **inputs,
    do_sample=True,
    top_p=0.9,
    temperature=0.9,
    max_length=1024,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

And a sample output:

### HUMAN:
What is artifical intelligence?

### RESPONSE:
Artificial intelligence, or AI, refers to the ability of computers to perform tasks that typically require human intelligence, such as decision making, problem solving, and language understanding. AI has been used in various fields, including healthcare, manufacturing, and finance, among others.

Further details

Please refer to the original model Griffin-3B.