The model has been quantized as part of the project GPTStonks. It works with transformers>=4.33.0 and it can run on a consumer GPU, with less than 3GB of GPU RAM. The libraries optimum, auto-gptq, peft and accelerate should also be installed.

Here is a sample code to load the model and run inference with it using greedy decoding:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "daedalus314/Marx-3B-V2-GPTQ"

tokenizer = AutoTokenizer.from_pretrained(model_id)
quant_model = AutoModelForCausalLM.from_pretrained(model_id, device_map='auto')

text = """### HUMAN:
What is artifical intelligence?

### RESPONSE:
"""
inputs = tokenizer(text, return_tensors="pt").to(0)

out = quant_model.generate(
    **inputs,
    max_length=1024,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))

And a sample output:

### HUMAN:

What is artifical intelligence?

### RESPONSE:

Artificial intelligence (AI) is a field of computer science that aims to imitate human intelligence. It is a subset of computer science that focuses on developing intelligent machines that can perform tasks that require human intelligence, such as learning, reasoning, problem-solving, and decision-making.

AI is a broad term that encompasses a variety of technologies, including machine learning, natural language processing, robotics, and computer vision. Machine learning is a subset of AI that focuses on the development of computer programs that can learn and improve their performance from experience without being explicitly programmed. Natural language processing is a subset of AI that focuses on the ability of computers to understand and process human language. Robotics is a subset of AI that focuses on the development of machines that can move, manipulate objects, and perform tasks similar to humans. Computer vision is a subset of AI that focuses on the ability of computers to understand and interpret images.

AI is being used in a wide range of applications, from self-driving cars to virtual assistants like Siri and Alexa. It has the potential to transform many industries, including healthcare, education, and manufacturing. AI is also being used in areas such as robotics, autonomous vehicles, and smart homes.

AI is a rapidly evolving field, and its capabilities are constantly expanding. As more data is collected and analyzed, AI systems are becoming more intelligent and capable of performing tasks that require human intelligence.

Further details

Please refer to the original model Marx-3B-V2.