Llama-3.1-8B-GPTQ-4bit-128g – AI Model by fmlini251 | AlphaNeural AI

You can deploy this model and start earning money today!

fmlini251/
Llama-3.1-8B-GPTQ-4bit-128g

like

0

text-generation

meta-llama/Llama-3.1-8B

Views

No views yet

Llama-3.1-8B GPTQ 4-bit (group size 128)

Built with Llama.

GPTQ-quantized meta-llama/Llama-3.1-8B, produced with AutoGPTQ 0.7.1.

Quantization settings


bits	4
group_size	128
damp_percent	0.01
desc_act	false
sym	true
true_sequential	true
static_groups	false
format	safetensors, fp16

Calibration

Dataset: wikitext / wikitext-2-raw-v1, train split
128 samples, sequence length 2048, seed 0
Sampling follows examples/quantization/basic_usage_wikitext2.py from AutoGPTQ

Usage

python

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

repo = "fmlini251/Llama-3.1-8B-GPTQ-4bit-128g"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoGPTQForCausalLM.from_quantized(repo, device="cuda:0", use_triton=False)

License

Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.

See LICENSE and USE_POLICY.md in this repository.