The goal: Make a medium quality quant of sophosympatheia/Magistry-24B-v1.1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM with KVO, accounting for system overhead.
The result: Mixed precision quantization of sophosympatheia/Magistry-24B-v1.1

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

Layout

Layer	Dims	Dims	Quant
token_embd	5120	131072.0	iq4_k

	blk	40
attn_k	5120	1024	iq6_k
attn_norm	5120	1	f32
attn_q	5120	4096	iq6_k
attn_v	5120	1024	iq6_k
ffn_down	32768	5120	iq4_k
ffn_gate	5120	32768	iq4_ks
ffn_norm	5120	1	f32
ffn_up	5120	32768	iq4_ks
attn_output	4096	5120	iq6_k

output	5120	131072	iq6_k
output_norm	5120	1	f32

Cheers

MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
sophosympatheia - for the merge effort.
Everyone whose finetunes were included in the merge!
bartowski - for the imatrix + the myriad of quants we all benefit from.

Koshkasa/sophosympatheia_Magistry-24B-v1.1-IQ4_KS.gguf

What's that?

Layout

Cheers

Koshkasa/
sophosympatheia_Magistry-24B-v1.1-IQ4_KS.gguf