AesSedai/

NVIDIA-Nemotron-3-Super-120B-A12B-GGUF

I uploaded the wrong splits for Q4_K_M / Q5_K_M and have corrected that now with the changes mentioned in the 03/11 update. Also added an IQ3_S quant now that there is a PR from @bartowski to fix the IQ4_NL quantization crash.

I've adjusted the Q4_K_M and Q5_K_M to use Q5_0 for the ffn_down_exps tensor, which brings the Q5_K_M quant size down substantially.

This repo contains specialized MoE-quants for NVIDIA-Nemotron-3-Super-120B-A12B-BF16. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.

This model is a little weird, architecturally. There isn't a ffn_gate_exps tensor in it, and the ffn_down_exps tensor has 2688 elements in it which means that it is not compatible with most Q*_K quantizations.

So you may notice that the ffn_down_exps here is a little odd, and producing an actual IQ3_S-sized quant like I normally do is tricky since the IQ4_NL quantization type is also not behaving well.

I've chosen to upload these 3 quants for now and hope that there will be some improvements soon.