You can deploy this model and start earning money today!

lmcoleman/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MagicQuant-GGUF

Name: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MagicQuant-GGUF
Availability: InStock

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Views

No views yet

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — MagicQuant hybrid GGUFs

Per-group hybrid quantizations of nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 (hybrid-SSM MoE, ~30B total / A3B active), produced by MagicQuant's measured evolutionary search: tensors are grouped by role, candidate per-group scheme assignments are actually rendered and perplexity-measured (not predicted), and winners are selected per size tier. imatrix-calibrated, KL-scored (kl_weight 0.1), 15 measured candidates over 3 rounds.

Pins: MagicQuant 14e9d62, Foundry 2f99202. Search ledger and the full pre-registered comparison record live with the run.

Files

file	size	PPL (wikitext-2, ctx 512, full corpus)	vs BF16 (6.83)
`...-Q4_K_M.gguf`	17.60 GiB	6.8948 ± 0.046	+0.95%
`...-Q5_K_M.gguf`	20.60 GiB	6.8579	+0.41%
`...-Q6_K.gguf`	24.13 GiB	6.7979	−0.47% (within noise of baseline)

All PPLs from one CPU llama.cpp instrument (ctx 512, full wiki.test.raw, same baseline). Q4 was re-measured standalone with error bars; Q5/Q6 values are the search's internal measurements on the identical instrument (the standalone Q4 re-measurement reproduced the search-internal value to 4 decimals).

Tier filenames follow effective-bpw bands, not a single uniform quant type — each file mixes schemes per group (e.g. the Q4 winner spends its budget as D:Q4_K_M E:Q8_0 H:BF16 K:BF16 O:BF16 Q:Q8_0 S:Q8_0 U:IQ4_NL X:MXFP4_MOE).

How the Q4 winner was validated

As a controlled experiment, an alternative per-tensor budget-knapsack allocation (MagicQuant v2, which predicts damage from a distortion table instead of measuring candidates) was built at exactly the Q4 winner's size, under identical pinned code, and measured on the same instrument:

At matched 17.6 GiB, the measured per-group search beat the predicted per-tensor allocation: +0.95% vs +3.47% quality loss — a +2.50% PPL gap, far outside the measurement error.

The losing artifact is not published (dominated at identical size); the numbers above are from the pre-registered comparison record.

Notes

Chat template embedded and verified present in every file before upload.
SSM/state tensors are held at F32/F16 per llama.cpp requirements (the never-quantize and f32-required-operand classes); BF16-designated groups are written as F16 on disk (llama.cpp BF16 compute-graph limitation).
Requires a llama.cpp recent enough for this architecture and for MXFP4_MOE/IQ4_NL tensor types (mid-2026 or newer).
Built and measured on an AMD Strix Halo (Ryzen AI MAX+ 395, gfx1151) host.

lmcoleman/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MagicQuant-GGUF

NVIDIA-Nemotron-3.5-Lightning-30B-A3B — MagicQuant hybrid GGUFs

Files

How the Q4 winner was validated

Notes

lmcoleman/
NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MagicQuant-GGUF