Views
No views yet
Llama-2-7b-hf quantized with AQLM using the 4-bit 4x8 scheme.meta-llama/Llama-2-7b-hfAQLM4x8 (4 codebooks, 8 bits per codebook)4-bit8convert_to_hf.py from the AQLM project, then exported with --save_safetensors and --save_tokenizer.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model_id = "dbw6/Llama-2-7b-AQLM-4Bit-4x8-hf"
4tokenizer = AutoTokenizer.from_pretrained(model_id)
5model = AutoModelForCausalLM.from_pretrained(
6 model_id,
7 torch_dtype="auto",
8 device_map="auto",
9 trust_remote_code=True,
10)