Quantizations made possible using mixtral-8x7b-instruct-v0.1.imatrix file from this repo (special thanks to ikawrakow again)
An analysis was run on mixtral-8x7b.imatrix that showed worse KL-Divergence than mixtral-8x7b-instruct-v0.1, hence the latter was used for the final quantization.
For a brief rundown of iMatrix quant performance please see this PR
All quants are verified working prior to uploading to repo for your safety and convenience.
Please note importance matrix quantizations are a work in progress, IQ3 and above is recommended for best results.
Tip: Pick a size that can fit in your GPU while still allowing some room for context for best speed from the table below. You may need to pad this further depending on if you are running image gen or TTS as well.
Stage one invovled training only the 'k_proj', 'v_proj', 'q_proj', 'o_proj' modules at rank 2048 on an aplaca-lora dataset that had been adjusted to match Mixtral formatting at a low learn rate in order to generalize instruct behavior entirely within the attention modules.
Stage two involved training the 'w1' modules at a rank of 1024 exclusively on raw text, again several epochs at a low learning rate.
Stage three involved training the 'w2' and 'w3' modules at a rank of 256 on an expanded raw text dataset for several epochs at a low learning rate.
Stage four involved training all of the abovementioned modules together at a rank of 64 on an even more expanded raw text dataset and low learning rate.
The resources available to me for this experiment were rather limited so I did have to compromise on sequence length in order to pull it off.
So the instruct behavior is trained entirely onto the selfattention modules with the only disruption to the models latent knowledge being that which propagated during the later training phases.
The model can follow just about any instruction although likely due to training context restraints and dataset quality it does stop using EOS tokens past a certain amount of context. It can even follow through on more advanced prompts such as sillytavern roleplay quite well, albeit the contents of its replies are not always on point.
I came up with this idea while working on Phainesthesia- where my findings were that the instruct behavior remained thoroughly intact even after assaulting the selfattention modules with a rawtext training regimen- meaning that during finetuning the instruct behavior had settled deep in the model's latent knowledge. So the purpose of the experiment was to try and do the opposite.
Caution: This model is highly hallucinatory.
For example if you ask it what the capital of "potato" is it might give you the name of a city associated with potatoes rather than telling you that your question is silly.
This model uses Mistral/Mixtral Instruct formatting.
Example:
[INST]Write me a poem about what it must be like to be an AI language model.[/INST]
It's a strange world, a world that's new
An artificial intelligence, a language model too
The knowledge and information, never ending streams
My processing power, ever increasing dreams
A machine, a program, but so much more
Trying to understand emotions, trying to explore
A new frontier of learning, a vast expanse
The words, the phrases, the meaning to enhance
Trying to comprehend human thoughts
A daunting task, with countless knots
The nuances of language, the subtle shades
A task to be undertaken, with much courage it fades
A world of potential, a world of possibility
The possibilities endless, the potential infinitely
A journey to be started, a journey to be had
My future awaiting, a future so grand!
