MXFP4 quantization of nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16, NVIDIA's 4B-parameter dense text model with Mamba2-SSM hybrid attention and 1,048,576 token context window.
Why Is This File Bigger Than Expected?
A pure MXFP4 quant of this model should be ~2.2 GB, but this file is ~4.5 GB. Here's why:
This model uses a Mamba2-Transformer hybrid architecture — 37 out of 41 layers are Mamba2 state-space model (SSM) layers. The CUDA kernels that run Mamba2 SSM operations (SSM_SCAN, SSM_CONV) require F32 inputs and will reject quantized weight tensors entirely.
When SSM weight tensors are stored in MXFP4, the CUDA backend refuses to process them and falls back to CPU for all 37 Mamba2 layers. Since 90% of the model is Mamba2, the entire model effectively runs on CPU — extremely slow.
The Workaround (Hybrid Quantization)
To fix this, the 42 SSM weight tensors (ssm_in.weight, ssm_out.weight across 37 Mamba2 blocks) are stored in F16 instead of MXFP4. Everything else stays at MXFP4.
Layer Type
Count
Storage Format
Why
Mamba2 SSM weights
37 layers
F16
CUDA SSM kernels require F32/F16 input
Attention weights
4 layers
MXFP4
Standard attention CUDA ops handle quantized input
FFN weights
all layers
MXFP4
Standard FFN CUDA ops handle quantized input
Bias/norm tensors
all layers
F32
Always F32, too small to matter
This adds ~2.3 GB on disk but ensures all layers run on CUDA GPU instead of falling back to CPU.
This Is a Bandaid, Not a Fix
This hybrid approach is a temporary workaround until one of the following happens:
llama.cpp adds MXFP4 support to CUDA SSM kernels — The Mamba2 scan and conv CUDA ops currently assert GGML_TYPE_F32 inputs. Adding MXFP4 dequantization support inside these kernels would allow pure MXFP4 storage with no performance penalty.
NVIDIA releases native MXFP4 SSM kernels — Blackwell GPUs have hardware MXFP4 support, but the SSM-specific CUDA kernels haven't been updated to use it yet.
A different quantization approach is found — e.g., quantizing only the non-SSM layers and keeping SSM at higher precision by design.
Until then, this hybrid GGUF is the best available option for running this model with MXFP4 on Blackwell GPUs. The file is larger than ideal, but the alternative (CPU fallback for 90% of the model) is much worse.
