A 14.33 GiB GGUF of NVIDIA-Nemotron-3.5-Lightning-30B-A3B that serves the full 262,144-token context on a single 16 GB GPU — with genuine IQ3_XXS expert tensors, which no stock build of this model can produce.

⚠️ This file requires a patched llama.cpp. Two separate reasons (see Building): the expert tensors are zero-padded, and this model needs upstream Dflash support that only landed on 2026-08-11.

Why this exists

Nemotron-3.5-Lightning's MoE expert tensors are 1856 / 2688 / 3712 wide. None is divisible by 256, so llama-quantize silently falls back to block-32 types for every block-256 (K/IQ) quant — the model's file size stops responding to the requested bit-width.

This is not theoretical. Reading the tensor census straight out of the published unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/…-UD-IQ3_XXS.gguf (19.76 GB):

tensor group	actual type
46 trunk expert tensors	IQ4_NL (block-32, ~4.5 bpw)
`blk.52` MTP experts	Q5_0
everything else	Q8_0 ×95, Q6_K ×30, F32 ×243, BF16 ×1
IQ3_XXS tensors	0

A file named IQ3_XXS that contains no IQ3_XXS at all. Every published sub-4-bit GGUF of this family has the same shape.

The fix: zero-pad the reduction axes so block-256 quants actually apply.

tensor	original	padded
`ffn_up_exps` reduction axis	2688	2816
`ffn_down_exps` (expert FF width)	1856	2048
shared-expert FF width	3712	3840

This is exact, not approximate: the MoE activation is relu², and relu²(0) = 0, so the padded rows contribute exactly zero. Zero is exactly representable in every ggml block format, so the padding stays inert after quantization.

Result: genuine IQ3_XXS experts, and the whole 262k context fits a 16 GB card.

Measured (RTX 5080 16 GB, sm_120, CUDA 13.0, WSL2)

metric	value
File	15,387,569,504 bytes (14.33 GiB)
VRAM @ 262,144 ctx, q4_0 KV, after load	14,784 MiB
VRAM @ full 250k-token prompt	14,878 MiB (1,100 MiB free)
Prefill @ 250k	3,819 t/s
Decode @ 250k depth	78.8 t/s
Generation smoke test	3/3 pass

Quality benchmarks (MBPP / HumanEval / agentic) have not been run on this build yet — see Status below. Nothing here claims a quality result.

sha256

0ba17723bce5f3e7faaf1941c6a3c393249f379c790fcb55d72e221095821aa8 Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf

Building llama.cpp (required)

Stock llama.cpp cannot load this file. You need current master (for Dflash) plus the padding patch included here.

bash

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
# Must include PR #26905 "Dflash support for nemotron-3.5" (merged 2026-08-11).
# Any master checkout after that date is fine.
git apply /path/to/nemotron-expert-padding.patch
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_ARCHITECTURES=120        # sm_120 = RTX 5080; use your own GPU's capability
cmake --build build --config Release -j

The patch is 3 files / 21 inserted lines: read the padded width from a GGUF key, size ffn_up_exps from it (in both the trunk and MTP paths), ggml_pad the MoE input to match, and size the per-expert output views from the down projection.

Toolchain caveat: NVCC 13.2 miscompiles the i-quant CUDA kernels (iq1_s/iq2_s/iq3_s fail test-backend-ops test -o MUL_MAT). This build was made with CUDA 13.0. If you are on 13.2+, rebuild against 13.0 before trusting IQ output.

Serving

bash

llama-server \
  -m Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf \
  --host 127.0.0.1 --port 8189 --alias lightning \
  -ngl 99 -np 1 --flash-attn on \
  -c 262144 -ctk q4_0 -ctv q4_0 \
  --jinja --temp 0.6 --top-p 0.95

serve-lightning.sh wraps this. Sampling, per NVIDIA's card for the family: 0.6 / 0.95 for tool calling (the served default above), 1.0 / 1.0 for reasoning — set per-request. Thinking is on by default; disable per-request with "chat_template_kwargs": {"enable_thinking": false}.

The MTP head

Lightning ships a multi-token-prediction head — blk.52, declared via block_count = 53 and nextn_predict_layers = 1. It is a full extra MoE layer (its own 128 experts).

It is retained in this file, with its experts at Q5_0. Two things to know:

It costs disk, not VRAM. llama.cpp sets mtp_flags = !load_mtp ? TENSOR_SKIP : 0, and load_mtp is false unless you pass --spec-type draft-mtp. On a normal load the server logs model has unused tensor blk.52.… -- ignoring for all 16 of them, and they are never allocated. The 14,878 MiB measured above is with the MTP head in the file.
Its experts cannot be quantized below ~4 bpw, by anyone. llama-imatrix only observes tensors on the normal forward path, and the MTP head isn't on it — so no importance data exists for blk.52, and llama-quantize correctly refuses ("Missing importance matrix … in a very low-bit quantization … bailing out"). Q5_0 is a block-32 type that needs none. This is the same approach unsloth shipped.

Speculative decoding via --spec-type draft-mtp is available but has not been tested here. If you try it, results are welcome in the discussions.

Do not try to shrink this by pruning layer 52: --prune-layers 52 updates block_count but not the per-layer array KVs, and the result fails to load with key nemotron_h_moe.feed_forward_length has wrong array length; expected 52, got 53.

Status / what's still open

imatrix — waiting on unsloth. This build uses bartowski's Lightning imatrix (padded to the padded geometry; included as imatrix-padded-lightning.gguf). Unsloth's own quant metadata references NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/imatrix_unsloth.gguf (unsloth_calibration_NVIDIA-Nemotron-3.5-Lightning-30B-A3B.txt, 80 chunks, 185 entries), but that file is not present in their repo — they published one for Nemotron-3-Nano but not for Lightning. We have asked. If it appears, this model will be rebuilt with it and the two compared directly.
Benchmarks not yet run on this build (MBPP / HumanEval / agentic).
MTP speculative decoding untested.

Files

file	purpose
`Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS.gguf`	the model
`nemotron-expert-padding.patch`	required llama.cpp patch (3 files, 21 lines)
`imatrix-padded-lightning.gguf`	bartowski's Lightning imatrix, padded to match
`pad_gguf.py`	pads a BF16 master (the technique)
`pad_imatrix.py`	pads an imatrix to the padded geometry
`verify_pad.py`	asserts the padded regions are exactly zero
`serve-lightning.sh`	the serve command above

Reproduction

bash

# 1. pad a BF16 master (e.g. unsloth's 2-shard BF16 GGUF)
python pad_gguf.py --variant b --in <master-*.gguf> --out lightning-padded-bf16.gguf
# 2. pad the imatrix to match
python pad_imatrix.py --variant b --in <imatrix.gguf> --out imatrix-padded-lightning.gguf
# 3. quantize with the patched build.
#    blk.52 rules MUST come first -- the first matching --tensor-type rule wins.
llama-quantize --imatrix imatrix-padded-lightning.gguf \
  --tensor-type "blk\.52\.ffn_up_exps=q5_0" \
  --tensor-type "blk\.52\.ffn_down_exps=q5_0" \
  --tensor-type ffn_up_exps=iq3_xxs \
  --tensor-type ffn_down_exps=iq3_xxs \
  lightning-padded-bf16.gguf out.gguf Q4_K_M 14

Provenance & licence

Base: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16, architecture nemotron_h_moe (Mamba2 + MoE hybrid, 128 experts, 52 trunk layers + 1 MTP layer, native 262,144 context).
Licence: NVIDIA Open Model License — see the base model card. Verify compliance for commercial use before redistributing.
The padding technique was developed 2026-08-10 for Nemotron-3-Nano-30B-A3B; see Nemotron-3-Nano-30B-A3B-pirola-IQ3_XXS-GGUF and the 12 GB variant Nemotron-3-Nano-30B-A3B-pirola-IQ2_XXS-XS-GGUF.
Upstream discussion on making this unnecessary: ggml-org/llama.cpp#27037.

pirola/Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS-GGUF

Nemotron 3.5 Lightning 30B A3B — padded IQ3_XXS (fork-only GGUF)

Why this exists

Measured (RTX 5080 16 GB, sm_120, CUDA 13.0, WSL2)

sha256

Building llama.cpp (required)

Serving

The MTP head

Status / what's still open

Files

Reproduction

Provenance & licence

pirola/
Nemotron-3.5-Lightning-30B-A3B-pirola-IQ3_XXS-GGUF