You can deploy this model and start earning money today!

OxxoCodes/
Meta-Llama-3-8B-Instruct-GPTQ

Name: Meta-Llama-3-8B-Instruct-GPTQ
Availability: InStock

Views

No views yet

OxxoCodes/Meta-Llama-3-8B-Instruct-GPTQ

Built with Meta Llama 3

Model Description

This is a 4-bit GPTQ quantized version of meta-llama/Meta-Llama-3-8B-Instruct.

This model was quantized using the following quantization config:

python

quantize_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,
    desc_act=False,
    damp_percent=0.1,
)

To use this model, you need to install AutoGPTQ. For detailed installation instructions, please refer to the AutoGPTQ GitHub repository.

Example Usage

python

from auto_gptq import AutoGPTQForCausalLM

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
model = AutoGPTQForCausalLM.from_quantized("OxxoCodes/Meta-Llama-3-8B-Instruct-GPTQ")

output = model.generate(**tokenizer("The capitol of France is", return_tensors="pt").to(model.device))[0]
print(tokenizer.decode(output))