bdambrosio/

Smaug-Mixtral-v0.1-8.0bpw-exl2

I know, I know, 6.5bpw is enough for perfection. For some people.

But, for those who want the best they can load, here's an 8bpw quant. Makes a difference for me, I think.

tweaked exl2 quant parameters a bit because I run 6-8k contexts:

python3 convert.py -i ../models/Smaug-Mixtral_v0.1 -o smaug_mixtral -cf smaug_mixtral -l 4096 -b 8 -hb 8 -ss 4096