You can deploy this model and start earning money today!

geninhu/
RakutenAI-7B-instruct-GPTQ

Name: RakutenAI-7B-instruct-GPTQ
Availability: InStock

Rakuten/RakutenAI-7B-instruct

quantized

apache-2.0

4-bit

Views

No views yet

RakutenAI-7B-instruct GPTQ

This is a 4-bit GPTQ quantized version of Rakuten/RakutenAI-7B-instruct.

Quantization Details

Method: GPTQ
Bits: 4
Group size: 128
Symmetric: True

Usage with vLLM

python

from vllm import LLM

llm = LLM(model="geninhu/RakutenAI-7B-instruct-GPTQ")

Usage with Transformers

python

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

model = AutoGPTQForCausalLM.from_quantized("geninhu/RakutenAI-7B-instruct-GPTQ")
tokenizer = AutoTokenizer.from_pretrained("geninhu/RakutenAI-7B-instruct-GPTQ")