AesSedai/

Mistral-Small-4-119B-2603-GGUF

This repo contains specialized MoE-quants for Mistral-Small-4-119B-2603. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.

I had made a Q4_K_M mix, but it kept returning NaN's for the KLD / PPL testing so I'm looking more into that.