Uniform NVFP4 weight-only (W4A16, group-16) quantization of
Nanbeige/Nanbeige4.2-3B. 3.6 GB,
the fastest artifact in the family (175 tok/s freeform / 220 summarize on RTX 5090) —
but read the gate table: use it for summarization/extraction-class work, NOT for
reasoning. The failures are published on purpose; they are the first FP4 datapoints for
a looped/weight-shared LLM anywhere (prior art, LoopQ arXiv:2605.16343, is INT-only).
Decode tok/s, single stream, this artifact's headline. Both rows measured
WITH ngram speculation (its intended copy-heavy workloads):
workload
spec decode
bf16
FP8-Dynamic
this repo
freeform
ngram, 8 tok
85
128
175
summarize / RAG (2k+ ctx prompts)
ngram, 8 tok
136
206
220
Two workload anchors, not an ISL sweep. The Marlin W4A16 kernel plus the
looped arch's double weight-read per token is where the speedup comes from:
weight compression pays out twice per token on this model.
Why reasoning breaks: this architecture executes each layer twice per token
(num_loops: 2) — 4-bit weight error compounds across both passes. Shallow extractive
work (summarization stays judge-tied with FP8 at perfect faithfulness) survives; long
reasoning chains don't. A contributing mechanism, verified by super-weight scan
(Apple 2411.07191): this model's super weight
(layers.1.mlp.down_proj.weight[1252, 6883], the single most load-bearing scalar)
sits in a 4-bit region under uniform quantization; the LoopShield recipe holds it
in FP8. A mixed-precision recipe recovers full reasoning parity at
4.5 GB — see the companion repo Nanbeige4.2-3B-NVFP4-FP8-LoopShield.
Serves via the Marlin W4A16 kernel (fastest batch-1 path on consumer Blackwell — do
not convert to W4A4: slower at batch 1 AND activation-quantization is this arch's
worst failure mode). Non-thinking mode recommended for its intended workloads:
chat_template_kwargs: {"enable_thinking": false}.
llmcompressor QuantizationModifier(targets="Linear", scheme="NVFP4A16", ignore=["lm_head"]), data-free. GPTQ was attempted and is not usable on this
architecture (llmcompressor 0.12: Hessian inversion fails on every module — root cause unknown; the double-fire hook was checked and is not it →
"Failed to invert hessian" on 154/154 modules, silent RTN fallback — reported
upstream); AWQ lacks arch mappings. So RTN is currently the only plain uniform-4-bit
route for looped models, which is precisely why the reasoning gate fails and why I
scope this artifact.
Limitations
Not for reasoning/agentic use — see gate table.
vLLM-only until PR #49433 merges; bundled modeling file carries two one-line
transformers-5 compat patches.
English-only evals, English-only calibration-free quant on a bilingual model.
add-on speculator for any of the above; thinking-aware retrain (2026-07-24), thinking-mode acceptance 0.41, lossless. Serve with TRITON_ATTN.
All three serve identically (same plugin, same flags); only the checkpoint differs.
Comparison chain: the columns here use my FP8-Dynamic quant as reference; the FP8 card carries the bf16-original matrix linking the chain back to the unquantized model.
Arch serving support: vLLM PR #49433 (until merged, the bundled vllm_plugin/ registers it out-of-tree)
Upstream tooling reports I filed from this work: llm-compressor#2952 (GPTQ Hessian inversion fails on all modules of this looped arch — root cause unknown; silent RTN fallback) · llm-compressor#2953 (AWQ lacks arch mappings)
Quantized with llm-compressor 0.12.0 (compressed-tensors format)
Benchmark provenance (ordered: public-harness first, then my closed harnesses)
Public, reproducible (lm-eval-harness local-chat-completions, exact configs in each row's annotation):
BBH — hard reasoning suite, CoT few-shot (non-thinking, 15/subtask)
MultiHop-RAG — multi-doc news QA; I run generator-only with gold evidence (custom harness, dataset public)
Closed/personal harnesses (not publicly reproducible — my own serving-workload gates; treat as relative signals between artifacts in THIS family, not cross-model scores):
Blind-judge summarization — 48 stratified real articles, per-article anonymized+shuffled candidates, single LLM judge scoring faithfulness/coverage/fabrication. Tests: does the quant change long-form grounded generation quality?
Rewrite taxonomy — 20 dictation-cleanup cases from a production ASR pipeline. Tests: instruction-constrained short-form editing.
JSON parse rate — structured-output emission over the summarization set. Tests: format discipline under quantization.
Citation
bibtex
1@misc{peciukonis2026nanbeige42nvfp4a16,
2 author = {Pe{\v{c}}iukonis, Matas (NullSense)},
3 title = {Nanbeige4.2-3B-NVFP4A16: uniform 4-bit quantization with published failure modes},
4 year = {2026},
5 howpublished = {Hugging Face},
6 url = {https://huggingface.co/NullSense/Nanbeige4.2-3B-NVFP4A16},
7 note = {Speed/summarization-scoped artifact; reasoning regression (GSM8K 89->81) documented on-card as a negative result for looped-arch uniform 4-bit.}
8}
