Quantization test results for NVIDIA-Nemotron-Nano-9B-v2, comparing standard llama.cpp quantization against a fork that adds two new non-linear block types: IQ2_NL and IQ3_NL.
The test model uses the nemotron_h hybrid Mamba-2 / attention architecture, whose tensor dimensions are divisible by 32 but not by 256. Every K-quant (Q2_K … Q6_K) and every I-quant (IQ1_M, IQ2_XXS, …) in llama.cpp operates on 256-element super-blocks. When a tensor's row length is not a multiple of 256, the quantizer cannot use the requested type and falls back to a 32-element block type.
NVIDIA-Nemotron-Nano-9B-v2 triggers this on almost every tensor in the model:
Tensor group
ncols
÷ 32
÷ 256
token_embd, output, attn_[q,k,v], ffn_up, ssm_in
4480
✅ 140
❌ 17.50
ffn_down
15680
✅ 490
❌ 61.25
attn_output
5120
✅ 160
✅ 20
ssm_out
10240
✅ 320
✅ 40
The consequence, taken directly from the quantization logs:
warning: blk.31.ffn_up.weight - ncols 4480 not divisible by 256 (required for type q2_K) -> falling back to q4_0
llama_model_quantize_impl: WARNING: 90 of 341 tensor(s) required fallback quantization
90 of 341 tensors, or rather 90 of the 122 quantizable tensors (73.8%) bypass the requested type entirely. And because the lowest-bpw 32-block type available in standard llama.cpp is Q4_0/IQ4_NL at 4.50 bpw, that fallback establishes a hard floor. A Q2_K quantization of this model does not produce a 2-bit model; it lands at ~4.50 bpw, and a IQ1_M lands at ~4.36 bpw. The sub-4 bpw region is unreachable.
The fork adds IQ2_NL and IQ3_NL non-linear quantization types built on the same layout as the existing IQ4_NL, extending the family downward.
Type
Block
Bits/weight
Bytes per 32 weights
Payload + scale
Origin
IQ2_NL
32
2.50
10
8 + 2
fork
IQ3_NL
32
3.50
14
12 + 2
fork
IQ4_NL
32
4.50
18
16 + 2
standard
Because these are 32-block types, nemotron_h's 4480 and 15680 geometries are no longer a barrier. Fallback count collapses from 90 → 1:
Build
Requested type
Fallbacks
Resulting BPW
standard
Q2_K
90 / 341
4.5000
standard
Q3_K
90 / 341
4.8400
fork
IQ2_NL
1 / 341
2.8046
fork
IQ3_NL
1 / 341
3.6722
The single remaining fallback is output.weight, requested as Q5_K but quantized to Q5_1 (6.00 bpw, 32-block).
1. Benefits
a) 32-block divisibility restores the low-bit type options
The 256-element super-block is a high-quality throughput optimisation and is an excellent option for architectures with power-of-two-friendly geometries, but a hard wall for everything else. nemotron_h and many recent architectures are the second case: neither d_model = 4480 nor d_ffn = 15680 is a whole number of super-blocks (17.5 and 61.25 respectively), so they fall back to other, usually sub-optimal, types.
IQ2_NL and IQ3_NL sidestep this entirely by inheriting IQ4_NL's 32-element block:
Fallbacks drop from 90 / 341 to 1 / 341.
The requested type is honoured on 120 & 121 of 122 quantizable tensors, where the naive Q2_K and Q3_K recipes honour only 27.
The delivered bit-rate finally tracks the requested bit-rate: IQ2_NL asks for 2.50 bpw and delivers 2.80 bpw overall (the overhead being FP32 norms, Q5_1 output, and IQ3_NL embeddings), where Q2_K asks for 2.6258 bpw and delivers 4.50.
The new types generalize well beyond Nemotron. Any architecture with non-power-of-two hidden dimensions like hybrid SSM/Mamba models, pruned or distilled checkpoints, models with unusual GQA head counts, and most width-searched Neural Architecture Search (NAS) derivatives hit the same wall, and the fix is the same.
b) bypassing the 4.5 bpw floor imposed by IQ4_NL
The complete set of 32-block quantization types in llama.cpp is:
Type
Bits/weight
IQ4_NL
4.50
Q4_0
4.50
Q4_1
5.00
Q5_0
5.50
Q5_1
6.00
Q8_0
8.50
The minimum is 4.50 bpw. For a model where the 256-block types are unavailable, that value is an absolute floor on the entire model. Every "aggressive" quant type collapses onto it: IQ1_M → 4.36 bpw, Q2_K → 4.50 bpw, Q3_K → 4.84 bpw. Three nominally distinct types spanning 1.75–3.44 bpw all land within 0.5 bpw of each other, at more than double the requested budget.
IQ2_NL (2.50) and IQ3_NL (3.50) extend the non-linear family below that floor:
Metric
Standard floor (IQ1_M)
IQ3_NL
IQ2_NL
BPW
4.3600
3.6722
2.8046
Size
4.52 GiB
3.81 GiB (−15.8%)
2.91 GiB (−35.7%)
Mean PPL
8.6505
8.3177
11.6323
Mean KLD
0.129217
0.078364 (−39.4%)
0.432011
Top-1 agree
83.879%
86.045%
71.111%
IQ3_NL wins on size and on every quality metric, but not on speed. It is smaller than IQ1_M on disk and simultaneously better on perplexity, KL divergence, token-probability RMS error, and top-1 agreement, but generates tokens ~35% slower (tg128 35.53 vs 54.70 t/s, pp1024+tg1024 66.38 vs 98.43 t/s). For a 9B model this moves the practical deployment target from "needs 5 GiB" to "fits in 4 GiB" and with better quality, provided that throughput cost is acceptable.
IQ2_NL is a valuable capability extension. At 2.80 bpw it produces a model 17.6% the size of F16, an option that previously did not exist at any quality, but the fidelity cost is substantial: PPL +50.28%, mean KLD 0.432, and top-1 agreement of only 71.1%. Roughly three tokens in ten change rank against the F16 reference. This is characteristic of ~2.5 bpw quantization generally rather than a defect of the type, and it is the expected shape of the curve; but it means IQ2_NL should be understood as a last-resort type for hard VRAM ceilings, not a default. IQ3_NL is where the fork's practical value sits.
Standard IQ1_M is outperformed once a coherent 2.5 bpw fallback exists.
Key Takeaways
IQ3_NL outperforms the entire sub-5-bpw range. At 3.81 GiB / 3.67 bpw it is 15.8% smaller than the smallest model standard llama.cpp can build (IQ1_M, 4.52 GiB / 4.36 bpw) while scoring materially better on every metric: PPL 8.3177 vs 8.6505, mean KLD 0.0784 vs 0.1292 (−39.4%), top-1 agreement 86.05% vs 83.88%.
IQ2_NL removes the 4.5 bpw floor. At 2.91 GiB / 2.80 bpw it is 35.7% smaller than anything the standard types can produce for this model.
IQ2_NL also outperforms IQ1_M. It is 7.8% smaller and 6.8% lower KLD than the fork's own IQ1_M.
Nothing at Q4_K and above changes. Those four quantized models are bit-for-bit identical across the two test runs; the quantization recipes never select the new types at those levels.
2. Disadvantages
Codebook efficiency. At 2.50 bpw, IQ2_NL sits at the same bit-rate as IQ2_S and above IQ2_XS (2.31) and IQ2_XXS (2.06), which spend their bits on lattice codebooks rather than a per-32 scale. On well-aligned models the super-block I-quants should still win at the low end. The NL family's value is availability, not peak efficiency.
Kernel maturity. Quantization wall-clock is ~2.7× the standard path (41.8 s vs 15.8 s for the 2-bit type on this test). More significantly, IQ3_NLinference is materially slower: tg128 35.53 t/s against 51.16 for standard Q3_K (−30.6%) and 54.70 for standard IQ1_M (−35.0%); pp1024+tg1024 66.38 against 92.00 and 98.43 (−27.8% and −32.6%). Prompt processing is unaffected (pp512 600.65 vs 600.77), which points at the token-generation dot-product path specifically. The regression does not generalise to the new types as a class: IQ2_NL posts the fastest tg128 in the benchmark table at 57.86 t/s (+2.3% over standard Q2_K) and the fork's IQ1_M is +2.1% over its standard counterpart, both while being substantially smaller. That isolates the cost to IQ3_NL's kernel rather than to 32-element blocks in general.
The mid-to-high types are untouched.Q4_K through Q8_0 are bit identical between the two runs. Those recipes still fall back to Q5_0/Q5_1/Q8_0, inflating Q4_K to 5.87 bpw and Q6_K to 8.22 bpw. Routing those fallbacks to IQ4_NL would shrink Q4_K on this architecture with no change to the new types.
output.weight remains a fallback. The recipe requests Q5_K and gets Q5_1 at 6.00 bpw for a 1120 MiB tensor (~420 MiB quantized), or ~14% of the entire IQ2_NL model. An IQ5_NL, or simply requesting IQ4_NL instead, would recover a meaningful fraction of that.
Single architecture, single model. Every conclusion here is drawn from one 9B hybrid-SSM checkpoint against one calibration corpus (imatrix). The 256-divisibility argument generalises well but the specific metrics results do not.
3. Comparative Performance Tables
Lower is better for every metric except Top-1.
Columns prefixed S· describe the standard (naive) build; columns prefixed F· describe the fork build. Rows 4–7 are identical between the two test runs.
S·Type
S·BPW
S·GiB
S·PPL
S·PPL Δ%
S·KLD
F·Type
F·BPW
F·GiB
F·PPL
F·PPL Δ%
F·KLD
IQ1_M
4.3600
4.52
8.6505
+11.76%
0.129217
IQ1_M
3.0416
3.15
11.8810
+53.50%
0.463744
Q2_K
4.5000
4.66
8.0878
+4.49%
0.052257
IQ2_NL
2.8046
2.91
11.6323
+50.28%
0.432011
Q3_K
4.8400
5.01
7.8840
+1.86%
0.025244
IQ3_NL
3.6722
3.81
8.3177
+7.46%
0.078364
Q4_K
5.8664
6.08
7.8034
+0.82%
0.006809
Q4_K
5.8664
6.08
7.8034
+0.82%
0.006809
Q5_K
6.3562
6.58
7.7752
+0.45%
0.003110
Q5_K
6.3562
6.58
7.7752
+0.45%
0.003110
Q6_K
8.2158
8.51
7.7488
+0.11%
0.000528
Q6_K
8.2158
8.51
7.7488
+0.11%
0.000528
Q8_0
8.5058
8.81
7.7471
+0.09%
0.000388
Q8_0
8.5058
8.81
7.7471
+0.09%
0.000388
As row 3 shows, IQ3_NL is 24.0% smaller than the naive Q3_K it replaces, but that naive Q3_K is a 4.84 bpw model wearing a 3-bit label.
Extended Metrics
Impl
Type
BPW
Size (GiB)
% of F16
Fallbacks
Mean PPL(Q)
Mean KLD
99% KLD
Δp RMS
Top-1 agree
-
F16
16.0040
16.57
100.0%
-
7.740272 ± 0.051484
-
-
-
-
naive
IQ1_M
4.3600
4.52
27.3%
90
8.650547 ± 0.058683
0.129217 ± 0.000514
0.904445
9.762%
83.879%
naive
Q2_K
4.5000
4.66
28.1%
90
8.087846 ± 0.053982
0.052257 ± 0.000208
0.354226
6.166%
89.151%
naive
Q3_K
4.8400
5.01
30.2%
90
7.883991 ± 0.052571
0.025244 ± 0.000102
0.167221
4.232%
92.203%
fork
IQ1_M
3.0416
3.15
19.0%
90
11.881048 ± 0.087265
0.463744 ± 0.001581
3.016266
18.739%
70.879%
fork
IQ2_NL
2.8046
2.91
17.6%
1
11.632334 ± 0.084906
0.432011 ± 0.001450
2.716625
18.216%
71.111%
fork
IQ3_NL
3.6722
3.81
23.0%
1
8.317650 ± 0.056391
0.078364 ± 0.000294
0.501990
7.755%
86.045%
both
Q4_K
5.8664
6.08
36.7%
90
7.803435 ± 0.052058
0.006809 ± 0.000026
0.044414
2.247%
95.761%
both
Q5_K
6.3562
6.58
39.7%
90
7.775161 ± 0.051827
0.003110 ± 0.000012
0.019825
1.500%
97.040%
both
Q6_K
8.2158
8.51
51.4%
90
7.748767 ± 0.051608
0.000528 ± 0.000002
0.002775
0.677%
98.633%
both
Q8_0
8.5058
8.81
53.2%
0
7.747085 ± 0.051616
0.000388 ± 0.000001
0.002175
0.598%
98.809%
Prompt Processing + Text Generation Benchmarks
Impl
Model
Size
Params
Backend
Threads
Test
T/S
standard
nemotron_h 9B IQ1_M - 1.75 bpw
4.51 GiB
8.89 B
MTL,BLAS
12
pp512
630.60 ± 0.78
standard
nemotron_h 9B IQ1_M - 1.75 bpw
4.51 GiB
8.89 B
MTL,BLAS
12
tg128
54.70 ± 0.08
standard
nemotron_h 9B IQ1_M - 1.75 bpw
4.51 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
98.43 ± 0.93
fork
nemotron_h 9B IQ1_M - 1.75 bpw
3.15 GiB
8.89 B
MTL,BLAS
12
pp512
625.29 ± 6.73
fork
nemotron_h 9B IQ1_M - 1.75 bpw
3.15 GiB
8.89 B
MTL,BLAS
12
tg128
55.86 ± 0.15
fork
nemotron_h 9B IQ1_M - 1.75 bpw
3.15 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
99.04 ± 0.92
standard
nemotron_h 9B Q2_K - Medium
4.66 GiB
8.89 B
MTL,BLAS
12
pp512
630.75 ± 5.27
standard
nemotron_h 9B Q2_K - Medium
4.66 GiB
8.89 B
MTL,BLAS
12
tg128
56.57 ± 0.26
standard
nemotron_h 9B Q2_K - Medium
4.66 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
102.88 ± 0.51
fork
nemotron_h 9B IQ2_NL - 2.5 bpw
2.90 GiB
8.89 B
MTL,BLAS
12
pp512
616.08 ± 10.77
fork
nemotron_h 9B IQ2_NL - 2.5 bpw
2.90 GiB
8.89 B
MTL,BLAS
12
tg128
57.86 ± 0.61
fork
nemotron_h 9B IQ2_NL - 2.5 bpw
2.90 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
98.73 ± 1.88
standard
nemotron_h 9B Q3_K - Medium
5.00 GiB
8.89 B
MTL,BLAS
12
pp512
600.77 ± 10.07
standard
nemotron_h 9B Q3_K - Medium
5.00 GiB
8.89 B
MTL,BLAS
12
tg128
51.16 ± 1.08
standard
nemotron_h 9B Q3_K - Medium
5.00 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
92.00 ± 0.61
fork
nemotron_h 9B IQ3_NL - 3.5 bpw
3.80 GiB
8.89 B
MTL,BLAS
12
pp512
600.65 ± 21.27
fork
nemotron_h 9B IQ3_NL - 3.5 bpw
3.80 GiB
8.89 B
MTL,BLAS
12
tg128
35.53 ± 0.50
fork
nemotron_h 9B IQ3_NL - 3.5 bpw
3.80 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
66.38 ± 0.58
standard
nemotron_h 9B Q4_K - Medium
6.07 GiB
8.89 B
MTL,BLAS
12
pp512
604.89 ± 0.46
standard
nemotron_h 9B Q4_K - Medium
6.07 GiB
8.89 B
MTL,BLAS
12
tg128
47.16 ± 0.25
standard
nemotron_h 9B Q4_K - Medium
6.07 GiB
8.89 B
MTL,BLAS
12
pp1024+tg1024
83.79 ± 0.76
Actual File Structure
What each recipe actually wrote to disk, across the 122 quantizable tensors:
The Q2_K row is the clearest illustration of the problem: 74% of the tensors in a "2-bit" model are stored at 4.5 bpw. The Q4_K and Q5_K rows show the fallback resolves upward to Q5_0 (5.50 bpw) and Q5_1 (6.00 bpw) against a Q4_K/Q5_K nominal of 4.50/5.50 bpw, inflating those builds by roughly 1.4 and 0.9 bpw respectively.
