NexesQuants/

MIstral-QUantized-70b_Miqu-1-70b-iMat.GGUF

Miqu 1 70b : a leak of Mistral Medium Alpha. Credit for this model goes to the Mistral AI company.

Miqu is probably the best 70b model I could ever play with, especially as a French speaker. Smart like a good Llama 2 70b finetune, no overfitting, little censorship, reasonable alignement, and even a sufficient mastery of French language to have a proper chat in French without feeling like spreaking to a broken bot.

Requantizations with iMatrix (better quality than without) of a Q5_K_M quant of a trending 70b model without better quant/fp16 available, this through a Q8_0 intermediary step.

Miqudev provided Q5_K_M, Q4_K_M, and Q2_K on this page : https://huggingface.co/miqudev/miqu-1-70b

Here, you will find the following quants :

Full offload possible on 48GB VRAM with a huge context size :

Full offload possible on 36GB VRAM with a variable context size (up to 7168 with Q3_K_M, for example)

Full offload possible on 24GB VRAM with a decent context size.

Full offload possible on 16GB VRAM with a decent context size.

Bonus : a Kobold.CPP Frankenstein which reads IQ3_XXS models and is not affected by the Kobold.CPP 1.56/1.57 slowdown at the cost of an absent Mixtral fix. https://github.com/Nexesenex/kobold.cpp/releases/tag/v1.57_b2030 Now supperseded with another KCPP-F, with 13 different KV cache quantization lebel to chose from : https://github.com/Nexesenex/kobold.cpp/releases

Miku 70b has a theta of 1,000,000, like CodeLlama, and not 10,000, like Llama 2 models usually have. That feature singularizes it to my knowledge to ALL Llama 2 models, beside Codellamas which also have a theta of 1,000,000..

-> So, no Alpha or Rope Base Frequency change is needed up to its base 32k context, if it works as intended. And if it does, no linear/yarn rope is necessary either to reach the base 32k context.

BUT Miqu is NOT a CodeLlama 70b (released only a few days after Miqu 70b), because :

So, CodeLlama 70b is nerfed like the other CodeLlama in general benchmarks terms, while Miku is matching a FINETUNED Llama-2 expectations.

Benchs I made with the original Q2_K quant of Miku 70b, most probably made from an initial FP16 and published by Miqudev :

A graph, courtesy of Ipechman, with the TQA of WinterGooddess 32k at 39.65728274 and not 20.

Data :

Miqudev's original Q2_K quant :

Benchs I made with the Q2_K I quantized from Miqudev's Q5_K_M with an intermediary Q8_0 step, and an iMatrix of 12800 tokens from wiki.train.raw :

Notice the perplexity drop brought by the iMatrix despite the requant.

Benchs I made with the Q3_K_M I quantized from Miqudev's Q5_K_M with an intermediary Q8_0 step, and an iMatrix of 12800 tokens from wiki.train.raw :

And now, the IQ3_XXS, new SOTA 3 bits quant from LlamaCPP, that I made in the same way :

Meanwhile, CodeLlama 70b Q2_K benches as such, to compare with Miqu 70B Q2_K originally quantized from FP16 by Miqudev :

And, for information, a comparable base Llama 2 70b finetuned by NousResearch for 32k context (Yarn) :

This yarn version performs closely to Llama 2 70b (but with 32k max context), and.. Much more poorly than Miqu 70b.

Also, for information, another requant from a Q4_K_S orphan of a 32k finetune of Sao10K's WinterGoddess 70b At Linear rope 2.5 (for 10k context) :

Draw your own conclusions as well !

New quants IQ1 :

V3 :

V5 :

CUSTOM QUANTS :

New quantizations strategies to bundle optimally the last tensor quantizations of Ikawrakow (LlamaCPP b2404 and beyond) and reach the hightest possible quality/size ratio. A work in progress. These quants strategies will be updated as soon as higher IQ tensor quants are available for the pertinent tensors.

Here's one new quant strategy, currently labelled IQ1_FSR, a 2% smaller quant than .IQ1_S v3/v5 with a quality slightly above IQ1_S_v3 but quite a bit lower than IQ1_S_v5 :

Second new quant strategy, currently labelled IQ1_PS, a 0.5% bigger quant than .IQ1_S v3/v5 with a quality a bit above with IQ1_S_v5 :

And 4 bigger IQ1 quants, up to 1.92 bpw :

IQ1_NS :

IQ1_MS :

IQ1_ES :

IQ1_SR :