This is a hybrid quantization of the language model using Q5_K_M. The embeddings are quantized to q8_0, the first three and last three layers are in q8_0, and the fourth layers from both the beginning and the end are in q6_K. Since the embeddings, as well as the first and last layers, are the most sensitive to quantization, I decided to quantize them less. This will allow achieving higher-quality responses from the AI at the cost of a slight increase in weight.
The quantization was done using llama.cpp version b8591 with this config:
