GGUF quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, built with llama.cpp ba360ef. Every claim below is measured; raw logs are in the metrics repo.

AD- marks an Atomic Dynamic layout: the two expert tensors are assigned explicitly instead of left to a preset, and the name carries the assignment — AD-<down_exps>-<up_exps>, collapsed to one name when both match. Everything outside the experts is Q8_0; it is 7% of the model, so spending bits there is cheaper than reasoning about it.

Files

Measured on eval_neutral held-out, KL divergence against the Q8_0 reference. tok/s is generation on a single RTX 5090, full offload.

File	down / up	Size	mean KLD	q99 KLD	top-1	tok/s
`…-BF16.gguf` (split)	bf16	65.9 GB	—	—	—	—
`…-Q8_0.gguf`	q8_0 / q8_0	35.0 GB	reference	—	—	290.6
`…-AD-Q5_0_XL.gguf`	q8_0 / q5_0	28.8 GB	0.00749	0.075	95.73%	304.1
`…-Q5_K_M.gguf`	q5_1 / q5_0 ¹	26.6 GB	0.01781	0.159	93.22%	338.7
`…-Q4_K_M.gguf`	q8_0 / q5_0 ¹	25.0 GB	0.02542	0.219	91.88%	348.5
`…-AD-IQ4_NL_L.gguf`	q5_1 / iq4_nl	22.4 GB	0.01345	0.123	94.16%	318.1
`…-AD-IQ4_NL.gguf`	iq4_nl / iq4_nl	19.7 GB	0.01728	0.157	93.49%	326.5
`…-imatrix.gguf`	—	55 MB	the calibration data
`dflash-…-BF16.gguf`	—	1.2 GB	speculative drafter
`dspark-…-BF16.gguf`	—	0.7 GB	see below — does not load

¹ what llama.cpp actually writes after its fallback rules, not what the preset asked for.

Pick AD-IQ4_NL unless you have a reason not to: it matches Q5_K_M on every metric while being 6.9 GB smaller, and it is the smallest quant on this model that still benefits from calibration.

Where the AD layouts win

	size	mean KLD
`AD-IQ4_NL`	19.7 GB	0.01728
`Q5_K_M`	26.6 GB	0.01781

Same quality, 6.9 GB less. And against ggml-org's NVFP4 build at effectively identical size:

	size	mean KLD	top-1
`AD-IQ4_NL_L`	22.4 GB	0.01345	94.16%
`NVFP4` (ggml-org)	22.5 GB	0.06233	86.75%

4.6× lower divergence at the same size. quantize_nvfp4 discards the importance matrix (GGML_UNUSED(quant_weights) in ggml-quants.c), so an NVFP4 file cannot be calibrated at all. That is the whole gap. NVFP4 does win on prompt processing — 14460 vs 11809 t/s at pp512, from native Blackwell FP4 kernels — so if you are prefill-bound rather than quality-bound it remains the better pick.

What we could not deliver: a usable 16 GB quant

Q2_0 (block 64, 2.25 bpw) is the only sub-4.5 bpw type that survives this model's shapes, and it is calibration-blind — quantize_q2_0 never forwards the imatrix. Measured, it does not hold up:

	size	mean KLD	top-1
`AD-Q2_0_L`	15.5 GB	1.097	53.0%
`AD-Q2_0`	11.4 GB	2.403	33.5%

A top-1 agreement of 33% means the model picks a different token two times out of three. These are published under experimental/ for reproducibility and should not be used for real work. There is currently no good 16 GB option for this model, from anyone — the smallest file here that we would stand behind is AD-IQ4_NL at 19.7 GB. Fixing that needs a block-64 superblock variant of the K-quant family in llama.cpp, which does not exist yet.

Why there is no 2-bit or 3-bit k/i quant

K- and I-quants store weights in superblocks of 256: 256 weights split into 16 sub-blocks of 16, each carrying a 6-bit scale and 6-bit minimum, all normalised against two fp16 super-scales. That hierarchy is what makes 2- and 3-bit quantization viable — at 2 bits you cannot afford an fp16 scale every 32 weights, since the scale alone would cost 0.5 bpw.

The price is that a tensor row must divide by 256. This model's expert tensors do not:

ffn_down_exps   ne0 = 1856 = 64 × 29      1856 % 256 = 64
ffn_up_exps     ne0 = 2688 = 128 × 21     2688 % 256 = 128

Together they are 93% of the model. When a row does not divide, llama.cpp's tensor_type_fallback() substitutes a block-32 type and logs a warning — it does not fail, and the file keeps the requested name.

requested	block	becomes	effective bpw
`IQ1_S` … `IQ3_S`, `IQ4_XS`	256	`IQ4_NL`	4.50
`Q2_K`, `Q3_K`	256	`Q4_0`	4.50
`Q4_K`	256	`Q5_0`	5.50
`Q5_K`	256	`Q5_1`	6.00
`Q6_K`	256	`Q8_0`	8.50

This is why published IQ2_XXS, IQ3_XXS, Q2_K and Q3_K_M files for this model all land around 18–19 GB, and why Q6_K and Q8_0 come out the same size. Every published IQ2_XXS for this model is a 4.5 bpw file.

Types that survive these shapes, and whether they read the imatrix:

type	bpw	block	imatrix
`Q1_0`	1.125	128	no — `up_exps` only; `down_exps` errors out
`Q2_0`	2.25	64	no
`MXFP4`	4.25	32	no
`Q4_0`	4.50	32	yes
`IQ4_NL`	4.50	32	yes
`NVFP4`	4.50	64	no
`Q5_0`	5.50	32	yes
`Q5_1`	6.00	32	yes
`Q8_0`	8.50	32	no (does not need it)

Not a defect in the model — NVIDIA's own NVFP4 uses group 16/64, which these shapes divide cleanly. It is specific to llama.cpp's k/i family.

To check any model before you quantize it:

bash

python3 -c "
import json,sys; c=json.load(open(sys.argv[1]))
for k in ('hidden_size','intermediate_size','moe_intermediate_size'):
    v=c.get(k)
    if v: print('%-22s %6d  %%256=%3d' % (k,v,v%256))
" config.json

Speculative decoding: all three paths, measured

The model supports three. MTP is inside every file here; DFlash and DSpark are sidecars.

bash

# MTP — no extra file needed
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  --spec-type draft-mtp -ngl 99 -c 8192

# DFlash — sidecar, auto-discovered by the dflash- prefix
llama-cli -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf \
  -md dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf \
  --spec-type draft-dflash -ngl 99 -ngld 99 -c 8192

Generation tok/s, single RTX 5090, full offload, identical prompt and seed, 400 tokens:

File	none	MTP	DFlash	MTP accept	DFlash accept
`AD-Q2_0`	358.4	278.1	169.0	77.9%	44.6%
`Q4_K_M`	348.5	236.5	161.5	71.6%	43.0%
`Q5_K_M`	338.7	214.1	159.8	65.4%	44.2%
`AD-IQ4_NL`	326.5	240.8	176.8	70.3%	50.4%
`AD-IQ4_NL_L`	318.1	232.2	160.8	69.5%	42.3%
`AD-Q5_0_XL`	304.1	226.5	160.3	72.1%	46.3%
`Q8_0`	290.6	189.5	143.1	61.4%	40.4%

On a fast GPU with full offload, speculation costs throughput on every quant — MTP lands at 65–78% of baseline, DFlash at 47–55%. Draft quality is not the problem: MTP accepts 61–78% of drafted tokens with a mean accepted length above 3.

The cost is the rollback. This is a hybrid Mamba2 model, so discarding a rejected draft means restoring SSM state, and the log shows 47.6 MiB copied per speculation step. We expected larger quants to flip the result, since the target gets more expensive per token while the checkpoint cost stays fixed. They did not — the ratio is flat across a 3× size range, so the copy scales with the work too.

Untested here, and where speculation should still pay: partial offload and CPU inference, where a token costs far more than 47.6 MiB of VRAM-to-VRAM copy. Measure on your hardware before enabling it.

DSpark does not load. llama.cpp requires a conf_proj.weight of shape {n_embd + dspark_markov_rank, 1} (src/models/dflash.cpp:94), but NVIDIA's DSpark checkpoint ships markov_head.markov_w1 / markov_w2 and no confidence projection, so it fails with check_tensor_dims: tensor 'conf_proj.weight' not found. The converted file is published anyway in case someone fixes the runtime side.

Converting either sidecar needs --target-model-dir, since neither carries a tokenizer:

bash

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash \
  --outtype bf16 --target-model-dir NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outfile dflash-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16.gguf

imatrix

Calibrated on AtomicChat/calib-corpora, recipe nemotron-3.5-lightning: 4.72M tokens over 3,035 documents — agentic 26.5%, code 19.1%, reasoning 15.9%, long-context 12.7%, multilingual 12.8%, vocabulary sweep 5.8%, structured 4.0%, graphics 3.2%.

Rendered through this model's own chat template, so <|im_start|>, <think>, <tool_call> and <tool_response> appear as the single special tokens the model actually sees. Run llama-imatrix with --parse-special, or that markup is tokenized as literal punctuation and ~40% of the corpus calibrates nothing.

checkpoint	mean cos	median	min	tensors still moving
512	0.9810	0.99924	0.4991	30
2000	0.9902	0.99906	0.7664	25
4000	0.9957	0.99977	0.8527	16
6000	0.9973	0.99989	0.8656	8
8512	0.99984	0.99999	0.9963	0
9008	0.99991	1.00000	0.9969	0

Past 8.5k chunks no tensor changes measurably, so more calibration data of this kind cannot change the quantizer's decisions. Reproduce with tools/converge.py in the dataset repo.

All 128 experts in all 46 expert tensors receive routing — 0 dead experts at every corpus size tested, including 823 chunks. Expert coverage is not what a larger corpus buys.

The MTP block (blk.52) is never executed in a normal forward pass, so it collects no imatrix data at any corpus size. It is pinned to q4_0 in every file: --tensor-type 'blk\.52\.=q4_0'.

Architecture

52 layers: 23 Mamba2, 23 MoE, 6 attention (indices 5, 12, 19, 26, 33, 42). 128 experts, 6 active plus 1 shared. hidden_size 2688, moe_intermediate_size 1856, squared-ReLU FFN — experts have up and down only, no gate. GQA 32/2, head_dim 128, rope_theta 10000, no sliding window, context 262144.

With only 6 attention layers at 2 KV heads the KV cache costs about 6 KB per token, and Mamba2 state is a constant ~50 MB per sequence regardless of context length. Long context is unusually cheap here — budget for weights, not for cache.

Running

llama-server -m NVIDIA-Nemotron-3.5-Lightning-30B-A3B-AD-IQ4_NL.gguf -ngl 99 -c 32768

Needs llama.cpp b10326 or newer for nemotron_h_moe. Prompt format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
<think>

Reproducing

bash

python3 convert_hf_to_gguf.py NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 \
  --outtype bf16 --outfile Nemotron-BF16.gguf

llama-imatrix -m Nemotron-Q4_0.gguf -f calib_train.txt -o imatrix.gguf \
  -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --save-frequency 500

llama-quantize --imatrix imatrix.gguf \
  --tensor-type 'blk\.52\.=q4_0' \
  --tensor-type 'ffn_down_exps=iq4_nl' \
  --tensor-type 'ffn_up_exps=iq4_nl' \
  Nemotron-BF16.gguf Nemotron-AD-IQ4_NL.gguf Q8_0 64

-b 8192 is not cosmetic: at the default batch the imatrix run takes 2h12m instead of 1h22m on the same hardware.

AtomicChat/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — GGUF

Files

Where the AD layouts win

What we could not deliver: a usable 16 GB quant

Why there is no 2-bit or 3-bit k/i quant

Speculative decoding: all three paths, measured

imatrix

Architecture

Running

Reproducing

AtomicChat/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF