AesSedai/

Nemotron-Cascade-2-30B-A3B-GGUF

This repo contains specialized MoE-quants for Nemotron-Cascade-2-30B-A3B. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.

This model is a little weird, similarly to the other recent Nemotrons. There isn't a ffn_gate_exps tensor in it, and the ffn_up_exps and ffn_down_exps tensors have 2688 elements in it which means that it is not compatible with most Q*_K quantizations.

Therefore, most of the quants have to use IQ4_NL, Q4_0/Q4_1, and Q5_0/Q5_1 quantizations for the FFNs.