the other 14 shannons will be remembered. HQQ quantized to 2 bits with 4 bit attention. Fits on a 3090 with room to grow. Supports full 32k context. I will not combine those assertions.
The attention tensors are 4 bit because mixtral reuses it for each expert - so it's only adding 0.4 GB and the quality improve dramatically. See this but horny and dying of chatml m<|alig>|nant tokenitis.|>
This is a 2+4 bit quantization of noromixmaidblah (just scroll down) using an emerging and aparrently very robust quantization method Half-Quadratic Quantisation. It ultimately squeezes it's tokens out of HF Transformers, not one ofthe lesser inference tools. So what's juicy about this is that it functions with full Transformers sampler and tokeniser support but you only need a 3090 instead of a H100! Truly emancipatory.
my best guess at parsing the HQQ source is that it works by sort of... 'JIT de-quanti-'' I have no idea, really. If you prefer talking to human beings from being lied to by language models (why are you here?) you could probably ask the MobiusML - they seem friendly and compsci/engineer types tend to enjoy talking about their research and development. Weirdos.
I think this is a functioning quant from one of everone's favorite norovirus inspired language models, Noromaid. I wouldn't know - I can't load 90 gigabytes of BF16 so this is my first few minutes too.
see my oom-killer nightmare log. (my struggle with baby's first quant) in the other markdown file.
note 5 mar 2024: oh yeah if this is happening there's a flag in the transformers function its calling to just make it stop? idk why it wasn't on by default.
But even if you do want to know what I've learned - you're better off just asking me than trying to parse that.
Just read the original card please:
Original README from the Neversleep twins follows:
license: cc-by-nc-4.0
image/png
Disclaimer:
This model is experimental, do not expect everything to work.
This model uses the Chatml prompting format
Beeg noromaid on steroids. Suitable for RP, ERP.
This model was trained on the Zloss fork of Charles, and should fix issue the model had.
Use Chatml prompt format, but not the special token.
The reason is that Axolotl merge the finetune with the base model at 1.0 weight basically, but this is too much, so I use another script available HERE to merge with less weight, sadly, it don't take the special Chatml token. It's like Orca2 for the matter.
Credits:
Undi
IkariDev
Description
This repo contains FP16 files of Noromaid-v0.4-Mixtral-Instruct-8x7b-Zloss.
