4-bit Quantized Model
This repository contains a high-efficiency 4-bit compressed version of the model. It is optimized using bitsandbytes (NF4 precision) to run on smaller graphics cards or consumer hardware.
Hardware Setup Used
Processed and generated on enterprise-grade hardware utilizing NVIDIA Multi-Instance GPU (MIG) architecture and high-capacity server systems.
Quick Start Code
To use this compressed version in your own Jupyter notebook, run:
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("YOUR_USERNAME/YOUR_REPO_NAME", device_map="auto")
