You can deploy this model and start earning money today!

rafiqrana/
llama-3.1-8B-Instruct-4bit

Name: llama-3.1-8B-Instruct-4bit
Availability: InStock

text-generation-inference

endpoints_compatible

4-bit

bitsandbytes

Views

No views yet

Use a pipeline as a high-level helper

python

from transformers import pipeline

pipe = pipeline("text-generation", model="rafiqrana/meta-llama-3.1-8B-Instruct-4bit")

Load model directly

python

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("rafiqrana/meta-llama-3.1-8B-Instruct-4bit")
model = AutoModelForCausalLM.from_pretrained("rafiqrana/meta-llama-3.1-8B-Instruct-4bit")

Simple Example

Install Dependencies

bash

pip install --upgrade transformers bitsandbytes accelerate

Run Example Code

python

import transformers
from huggingface_hub import login

ACCESS_TOKENS='YOUR_ACCESS_TOKEN_HERE'

login(ACCESS_TOKENS)

model_id = "rafiqrana/llama-3.1-8B-Instruct-4bit"

pipeline = transformers.pipeline(
    task="text-generation",
    model=model_id,
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a intelligent assistent."},
    {"role": "user", "content": "Introduce yourself concisely."},
]

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

while True:
    response = pipeline(
        messages,
        max_new_tokens=256,
        eos_token_id=terminators,
    )

    print(response[0]["generated_text"][-1]["content"])
    messages.append({"role": "user", "content": input("Ask me anything >> ")})