4 bit quantized version of Llama-3.1-8B
Created via:
from huggingface_hub import upload_folder
#Load the pre-quantized model and tokenizer
quantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16 )
#Load the quantized model from Colab environment
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", quantization_config=quantization_config, device_map='auto')
