You can deploy this model and start earning money today!

MISHANM/
meta-llama-8B-int8

Name: meta-llama-8B-int8
Availability: InStock

safetensors

llama

meta-llama/Meta-Llama-3-8B-Instruct

quantized

8-bit

bitsandbytes

Views

No views yet

MISHANM/meta-llama-8B-int8

This model is an INT8 quantized version of the meta-llama/Meta-Llama-3-8B-Instruct, designed for enhanced performance on supported hardware with a balance between speed and accuracy.

Model Details

Tasks: Causal Language Modeling, Text Generation
Base Model: meta-llama/Meta-Llama-3-8B-Instruct
Quantization Format: INT8

Device Used

GPUs: 1*AMD Instinct™ MI210 Accelerators

Inference with HuggingFace

python3

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the fine-tuned model and tokenizer
model_path = "MISHANM/meta-llama-8B-int8"

model = AutoModelForCausalLM.from_pretrained(model_path,device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_path)

# Function to generate text
def generate_text(prompt, max_length=1000, temperature=0.9):
   # Format the prompt according to the chat template
   messages = [
       {
           "role": "system",
           "content": "Give response to the user query.",
       },
       {"role": "user", "content": prompt}
   ]

   # Apply the chat template
   formatted_prompt = f"<|system|>{messages[0]['content']}<|user|>{messages[1]['content']}<|assistant|>"

   # Tokenize and generate output
   inputs = tokenizer(formatted_prompt, return_tensors="pt")
   output = model.generate(  # Use model.module for DataParallel
       **inputs, max_new_tokens=max_length, temperature=temperature, do_sample=True
   )
   return tokenizer.decode(output[0], skip_special_tokens=True)

# Example usage
prompt = """Give a poem on LLM ."""
text = generate_text(prompt)
print(text)

Citation Information

@misc{MISHANM/meta-llama-8B-int8,
  author = {Mishan Maurya},
  title = {Introducing INT8 quantized version of meta-llama/Meta-Llama-3-8B-Instruct},
  year = {2024},
  publisher = {Hugging Face},
  journal = {Hugging Face repository},
  
}