What's that?
The goal: Make a quality quant of ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.
This time the wheel was reinvented one spoke at a time - blocks in proximity to input and output are treated with premium precision, while the more tolerant middle blocks are compressed in a more aggressive 4-bit precision. Attention is kept high in all blocks.
This is in line of how popular Q4_K_S/M quants are made for Mistral 24B, except we use IQ_K quants and utilize trellis for middle ffn_up and ffn_gate blocks.
Three versions were cooked. Two lost marginally to Q4_K_S in hellaswag and winogrande, though perhaps they would perform better in NIHS-tests (due to higher attn), which I don't have the patience to set up. For what they offered, they were slow - they had even more trellis, and it shows.
Option three, which is this one, compromised some attn precision for better protection of ffn_down layers. Lacking a conventional naming scheme for these, I went with IQ4_K_M. It looks stupid. I don't know how to fix that.
The result: Mixed precision quantization of
ShyliaSafetensors/Ariel-Alloy-V1-24B-Heretic
quantized with ik_llama.cpp build: 9d07d868
incompatible with mainline llama.cpp
Layout: IQ4_K_M
| Layer | Dims | Dims | Quant |
|---|
| token_embd | 5120 | 131072.0 | iq5_ks |
Blocks 0, 1, 38, 39
| Layer | Dims | Dims | Quant |
|---|
| attn_k | 5120 | 1024 | iq6_k |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 4096 | iq6_k |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 32768 | 5120 | iq6_k |
| ffn_gate | 5120 | 32768 | iq6_k |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 32768 | iq6_k |
| attn_output | 4096 | 5120 | iq6_k |
Blocks 2, 3, 37, 34-37
| Layer | Dims | Dims | Quant |
|---|
| attn_k | 5120 | 1024 | iq6_k |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 4096 | iq5_k |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 32768 | 5120 | iq5_k |
| ffn_gate | 5120 | 32768 | iq5_ks |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 32768 | iq5_ks |
| attn_output | 4096 | 5120 | iq5_k |
Blocks 6–33
| Layer | Dims | Dims | Quant |
|---|
| attn_k | 5120 | 1024 | iq5_k |
| attn_norm | 5120 | 1 | f32 |
| attn_q | 5120 | 4096 | iq5_ks |
| attn_v | 5120 | 1024 | iq6_k |
| ffn_down | 32768 | 5120 | iq4_k |
| ffn_gate | 5120 | 32768 | iq4_kt |
| ffn_norm | 5120 | 1 | f32 |
| ffn_up | 5120 | 32768 | iq4_kt |
| attn_output | 4096 | 5120 | iq5_k |
| Layer | Dims | Dims | Quant |
|---|
| output | 5120 | 131072 | iq6_k |
| output_norm | 5120 | 1 | f32 |
Rationale
Hopefully the beefed up attention will help over contexts this quant is intended to run (16k-32k). It is not statistically dumber than its main competitor, Q4_K_S, outsmarted by 3-4 responses over 1267 winogrande and 800 hellaswag questions (s = 123). While the difference is not statistically significant, it's there. A NIHS test would probably be this quant's stronger suit, but I lack quality data to test it. Experimental quant. WYSIWYG.
Cheers
MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
ShyliaSafetensors - for the merge effort.
Everyone whose finetunes were included in the merge!
mradermacher - for the imatrix + the myriad of quants we all benefit from.
