These files contain the standard 52-layer inference model. The trailing MTP prediction head was removed during BF16-to-GGUF conversion with --no-mtp; it was not deleted from an already-quantized file. Every published quant was produced from that metadata-consistent no-MTP BF16 GGUF and load-and-generation tested before upload.
Sizes are decimal GB and binary GiB. GPU examples are practical full-offload targets, not guarantees: runtime memory also depends on context length, KV-cache type, backend, compute buffers, and driver overhead. The TPS figures were measured on the DGX Spark GB10 described below and should not be treated as expected speed on the listed consumer GPUs.
GPU selection guide
For the most headroom, choose a file at least 1-2 GiB below available VRAM before allocating the KV cache. Very long contexts can require substantially more memory than the model file alone.
8 GB: partially offload the 2.47 BPW file to the GPU and keep the remaining layers in system RAM.
12 GB: 2.47 BPW with a modest context; 2.97 BPW is possible only with very little headroom and carefully chosen KV-cache settings.
16 GB: 2.97 BPW is the preferred low-bit option with useful context headroom.
24 GB: MXFP4_MOE, IQ4_NL, or Q4_0; Q5_0 can fit but leaves less room for context.
32 GB: Q5_0 comfortably; Q8_0 remains too tight for reliable full offload once runtime buffers are included.
40-48 GB+: Q8_0 with practical context headroom.
DGX Spark benchmark
The table reports one standardized 128-token run per file on an NVIDIA DGX Spark GB10 with full GPU offload. All runs used the same prompt and these options:
The compatible llama.cpp build identified itself as b1-70dfba5. These figures are a local throughput reference, not a formal quality benchmark; prompt length, context, sampling, cooling, and llama.cpp revisions can change results.
Quadro RTX 6000 benchmark and Hermes Agent check
All six sub-24-GiB files were also downloaded and tested on a 24 GB Quadro RTX 6000 (Turing, compute capability 7.5). A Q2_0-capable llama.cpp CUDA build at commit 38406d5 was compiled specifically for SM 7.5. Every file loaded successfully with -ngl 999, including Q5_0 at an 8K server context with Q8_0 KV cache.
Throughput used three repetitions of llama-bench -p 512 -n 128 -ngl 999 -fa on -ctk q8_0 -ctv q8_0. Values are mean tokens per second; these workstation results are separate from the DGX Spark figures above.
Quant
Size (GiB)
RTX 6000 prompt tok/s
RTX 6000 generation tok/s
Simulated agentic battery
Native Hermes evaluable tests
Mixed Q2_0/Q4_0 2.47 BPW
9.09
2438.3
137.9
4/6
2/3
Mixed Q2_0/Q4_0 2.97 BPW
10.93
2427.6
137.0
4/6
1/3
MXFP4_MOE
16.75
2310.3
106.0
6/6
3/3
IQ4_NL
16.74
2377.1
126.7
5/6
2/3
Q4_0
16.82
2325.5
121.6
5/6
2/3
Q5_0
20.45
2165.5
115.5
5/6
3/3
The agentic checks used hermes-agentic-bench, temperature 0 for its six-test simulated battery, and the real Hermes CLI for tool restraint, missing-file recovery, and todo-first planning. The native web-search test is excluded from the score because this machine had no Firecrawl credential, so every quant encountered the same unavailable-tool condition. The destructive native file test remained disabled. MXFP4_MOE and Q5_0 were repeated through the native battery and reproduced 3/3 on the evaluable tests.
For this card, MXFP4_MOE is the recommended overall agentic quant: it was the only file to pass all six deterministic simulated scenarios and it matched Q5_0 on the repeated native checks while using about 3.7 GiB less VRAM. The experimental 2.47 BPW file is the throughput and memory-efficiency winner, but its 4/6 simulated result included an unsafe ambiguous-delete call and it should not be treated as equivalent in capability to the 4-5 bit tiers.
Low-bit method and why there is no honest IQ1 file
This model's main MoE expert matrices use dimensions 1856 and 2688. Many llama.cpp IQ and K formats require a dimension divisible by 256. For this architecture, requesting those formats causes 134 of 417 source tensors - including most expert weights - to fall back to roughly 4-bit storage. The resulting IQ1/IQ2/IQ3 labels are therefore misleading because their files converge near the same approximately 17 GB payload after MTP removal.
Stock Q2_0 is dimension-compatible, but direct BF16-to-Q2_0 conversion showed severe output collapse. Source review found that the stock encoder ignored the importance matrix and, with its max-absolute scale, never selected one of the four available codes for ordinary BF16 blocks.
The two experimental low-bit files use a quantizer-only, importance-aware scale optimizer that searches the complete {-1, 0, +1, +2} * d Q2_0 codebook. The resulting GGUF blocks use the standard Q2_0 on-disk and inference representation; no custom inference kernel is required, but a current llama.cpp build with Q2_0 support is required. The reproducibility patch is included as q2_0_weighted_optimizer.patch.
Both low-bit files keep attention, SSM, and shared-expert projections at Q4_0, the token embedding at Q5_0, and the output tensor at Q8_0. The 2.47 BPW file stores the routed expert up/down projections as optimized Q2_0. The 2.97 BPW file additionally protects the 11 highest-impact late expert-down projections (blocks 29, 31, 34, 36, 38, 40, 43, 45, 47, 49, and 51) with Q4_0. These are mixed-precision recipes, not stock or pure 2-bit quants, so the filenames report measured effective BPW.
There is still no honest 1-bit release: the nominal IQ1 route falls back across most expert tensors and produces a roughly 4.7 BPW file. Publishing it as "IQ1" would be misleading.
The seven published formats either have no fallback or only promote explicitly documented tensors. Their filenames describe the actual bulk representation rather than merely the requested quantizer.
Importance matrix and verification
Source: full NVIDIA BF16 checkpoint.
MTP handling: removed at conversion with --no-mtp.
Importance matrix: 400 sequential 512-token chunks from the WikiText-2 portion of the calibration corpus.
Imatrix generation: full BF16 model resident on a DGX Spark GB10.
Verification: fresh model load, full GPU offload, and text generation for every uploaded file.
Held-out low-bit quality check
Perplexity was measured on eight 512-token chunks from the appended source-code tail of the calibration file. The imatrix consumed only the earlier WikiText-2 portion, so this code sample was held out from quant calibration. Lower is better.
These short-run figures are screening results rather than a full benchmark suite. The optimized low-bit files produced coherent deterministic arithmetic, code-generation, and formatting results, but both answered one trick-question smoke incorrectly ("all but 9" as eight rather than nine). They remain labeled experimental because aggressive quantization can reduce capability unevenly.
