The goal: Make a competitive 4-bit quality quant of TheDrummer/Big-Alice-28B-v1 using SOTA quant types from ik_llama.cpp, allowing the resulting gguf to fit into 16gb VRAM without KVO, accounting for system overhead.
The result: Mixed precision quantization of TheDrummer/Big-Alice-28B-v1 utilizing 4-bit trellis quants for FFN tensors that is theoretically smarter and faster than IQ4_XS.

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp

Layout

Layer	Dims	Dims	Quant
token_embd	5120	131072.0	iq4_k

	blk	40
attn_k	5120	1024	iq5_ks
attn_norm	5120	1	f32
attn_q	5120	4096	iq5_ks
attn_v	5120	1024	iq6_k
attn_output	4096	5120	iq5_k
ffn_down	14336	5120	iq4_kt
ffn_gate	5120	14336	iq4_kt
ffn_norm	5120	1	f32
ffn_up	5120	14336	iq4_kt

output	5120	131072	iq6_k
output_norm	5120	1	f32

Conclusion

In retrospect, it probably wasn't the best idea for 16GB VRAM - the PCI-E travel overhead from KV cache in RAM is MASSIVE for upscaled Apriel (especially for my 8-lane 5060 Ti.). This one remains an ok option at 20gb+ VRAM, or on 16gb GPUs with a less impotent PCI-E config. A smaller 3.44 bpw ffn + trellis 4.0 bpw attention quant is in the overn for strictly 16gb inference.
Regardless, this one is provided as is. WYSIWYG.

Cheers

SillyTilly - for the base model.
TheDrummer - for the finetuned model.
mradermacher - for the imatrix + the myriad of quants we all benefit from.
ikawrakow and contributors of ik_llama.cpp - I probably misused your ingenious creation.

Koshkasa/TheDrummer_Big-Alice-28B-v1-IQ4_KT-GGUF

What's that?

Layout

Conclusion

Cheers

Koshkasa/
TheDrummer_Big-Alice-28B-v1-IQ4_KT-GGUF