Quantization made by Richard Erkhov.
This is an upscaled fp16 variant of the original CodeLlama-13b-instruct base model by Meta after it has been loaded with nf4 4-bit quantization via bitsandbytes.
The main idea here is to upscale the linear4bit layers to fp16 so that the quantization/dequantization cost doesn't have to paid for each forward pass at inference time.
1import torch
2from transformers import AutoModelForCausalLM
3
4model = AutoModelForCausalLM.from_pretrained(
5 "arnavgrg/codellama-13b-instruct-nf4-fp16-upscaled",
6 device_map="auto",
7 torch_dtype=torch.float16,
8)
