This quant was made using exllamav2-0.2.7 with default dataset and extended quantization sample length (8k instead of default 2k). It also uses -head_bits=8 and max accuracy quant for first and last layer (8bpw), all other layers of the model use normally chosen methods (method and name (6.5bpw_L) inspired by quants like Q4_K_L and Q6_K_L made by
bartowski)
It fits nicely in 24GB VRAM on Windows with 20k fp16 context (should fit all 32k that with q8 cache in exl2).
